Brain Harmony: A Multimodal Foundation Model Unifying Morphology and Function into 1D Tokens
Authors: Zijian Dong
Li Ruilin
Joanna Su Xian Chong
Niousha Dehestani
Yinghui Teng
Yi Lin
Zhizhou Li
Yichi Zhang
Yapei Xie
Leon Qi Rong Ooi
B.T. Thomas Yeo
Juan Helen Zhou
Venue: NeurIPS 2025 poster ()
Keywords: Multimodal Neuroimaging; Brain Foundation Model; 1D Tokens; Representation Learning; Neuropsychiatric Disorders; Cognition
Primary area: neuroscience_and_cognitive_science
TL;DR: Brain Harmony (BrainHarmonix) unifies T1 MRI + fMRI into 1D tokens and tops six brain-disorder benchmarks.

Abstract: We present **Brain Harmony  (BrainHarmonix)**, the first multimodal brain foundation model that unifies structural morphology and functional dynamics into compact 1D token representations. The model was pretrained on two of the largest neuroimaging datasets to date, encompassing 64,594 T1-weighted structural MRI 3D volumes (～ 14 million images) and 70,933 functional MRI (fMRI) time series. BrainHarmonix is grounded in two foundational neuroscience principles: *structure complements function* - structural and functional modalities offer distinct yet synergistic insights into brain organization; *function follows structure* - brain functional dynamics are shaped by cortical morphology. The modular pretraining process involves single-modality training with geometric pre-alignment followed by modality fusion through shared brain hub tokens. Notably, our dynamics encoder uniquely handles fMRI time series with heterogeneous repetition times (TRs), addressing a major limitation in existing models. BrainHarmonix is also the first to deeply compress high-dimensional neuroimaging signals into unified, continuous 1D tokens, forming a compact latent space of the human brain. BrainHarmonix achieves strong generalization across diverse downstream tasks, including neurodevelopmental and neurodegenerative disorder classification and cognition prediction - consistently outperforming previous approaches. Our models - pretrained on 8 H100 GPUs - aim to catalyze a new era of AI-driven neuroscience powered by large-scale multimodal neuroimaging.

Brain Harmony: A Multimodal Foundation Model Unifying Morphology and Function into 1D Tokens
Zijian Dong"l
Ruilin Li
Joanna Su Xian Chong' Niousha Dehestani! Yinghui Teng' B.T. Thomas Yeo
Yi Lin'
Zhizhou Li
Yichi Zhang'
Yapei Xie Leon Qi Rong Ooi!
Juan Helen Zhout!
National University of Singapore Now MiroMind Al
Abstract
We present Brain Harmony (BrainHarmoonix), the first multimodal brain foun- dation model that unifies structural morphology and functional dynamics into compact ID token representations. The model was pretrained two of the largest neuroimaging datasets to date, encompassing 64,594 Tl-weighted structural MRI 3D volumes (-14 million images) and 70,933 functional MRI (fMRI) time ries. BrainHarmonix is grounded in two foundational neuroscience principles: structure complements function structural and functional modalities offer dis- tinet yet synergistic insights into brain organization; function follows structure brain functional dynamics are shaped by cortical morphology. The modular pre- training process involves single-modality training with geometric pre-alignment followed by modality fusion through shared brain hub tokens. Notably, our dy- namics encoder uniquely handles fMRI time series with heterogeneous repetition times (TRs), addressing major limitation in existing models. BrainHarmonix is also the first to deeply compress bigh-dimensione neuroimaging signals into unified, continuous ID tokens, forming compact latent space of the human brain. BrainHarmonix achieves strong generalization across diverse downstream tasks, including neurodevelopmental and neurodegenerative disorder classification and cognition prediction consistently outperforming previous approaches. Our models pretrained on H100 GPUs aim to catalyze new era of Al-driven neuro- science powered by large-scale multimodal neuroimaging. Code is available at: https: //github. com alb/ BBra/rrnnn-iee Brriinaaarooony
1 Introduction
The human brain is an extraordinarily complex organ, characterized by intricate anatomical archi- tecture and dynamic functional processes. To investigate these aspects in vivo, researchers rely on neuroimaging techniques that probe brain structure and function. However, each neuroimaging modality captures only single facet of this multifaceted system 2 3 This limitation highlights the necessity of multimodal neuroimaging approaches that combine complementary information (e.g., both structural and functional) to offer a holistic understanding of human cognition and improve clinical applications.
Recent advances in brain foundation models have transformed Artificial Intelligence (AI) for neu- roimaging analysis from task-specific approaches to self-supervised pretrained models capable of
"Equal contribution
Corresponniig author: helen. zhou@mus. edu. .gg
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(Morphologica) Structure)
Neuredevellopment Disorders
M
Brain Hob
Cognition
Harmonics
Blological Constraint
BrainHarmonix
Pre-align
Neurodegeneration
Disorder
mmm
(Dynamics with heterogenecus TR)
Figure I: Overview of Brain Harmony (BrainHarmonix), Brain morphology from TI Te-weihhted MRI (sMRI) and functional dynamics from fMRI are unified into compact ID brain-hub tokens, which can be readily adapted to downstream tasks via an attached projection head. Specifically, functional dynamics are pre-aligned with group-level geometric harmonics, with built-in flexibility handle heterogeneous repetition times (TRs). This fusion creates compact yet expressive representation space that effectively captures the interplay between brain structure and function, supporting a broad range of downstream applications, including neurodevelopmental and neurodegenerative disorder classification and cognition prediction.
adaptation across diverse downstream applications 4 6 7 Despite their promising generalizabil- ity, these models focus on either brain structure 6 (e.g., TI, T2- we-weietted MRI) function 4 5 (e.g., functional MRI (fMRI), without capturing the two complementary aspects simultaneously. Furthermore, recent neuroscience findings demonstrate that brain activity can be formulated exci- tations of fundamental resonant modes shaped by the brain's geometry, revealing how morphological structure fundamentally constrains functional dynamics [8]. Nevertheless, existing brain dynamics foundation models overlook this crucial constraint imposed by brain morphology. On the other hand, existing brain dynamics foundation models rely exclusively on fMRI datasets with homogeneous temporal resolutions [4] 5). hindering the integration of datasets collected from diverse scanners and protocols with varying repetition times (TRs). Even within individual datasets or real-world clinical scenarios, multiple TRs often coexist (9 10 [I] rendering previous models infeasible for broader deployment. This limitation substantially reduces available sample sizes and constrains comprehensive modeling of brain dynamics across multiple temporal scales. While connectivity- based approaches are naturally agnostic to variations in TR [7], they aggregate activity across entire scanning sessions, discarding essential non-stationary dynamics (e.g., transient state transitions and evolving co-activation patterns) in the blood-oxygen-level dependent (BOLD) signals [12
13 4 5)
Together, the aforementioned gaps highlight a fundamental challenge: creating comprehensive brain representations that effectively capture both structural and functional neuroimaging data with heterogeneous temporal resolutions. A critical step toward addressing this challenge is developing efficient methods to compress high-dimensional neuroimaging data into compact, information- dense representations. Transforming complex neuroimaging data into sequential 1D tokens offers promising solution, potentially providing a unified framework for integrating multimodal information across diverse neuroimaging acquisitions.
In this paper, we propose Brain Harmony (BrainHarmonix) address these critical gaps (Figure Db. Our major contributions include: (1) Developing the first multimodal brain foundation model to bridge morphological structure and functional dynamics in compact, information-rin representation space with 1D tokens. (2) Incorporating geometric harmonics to pre-align cortical morphology and functional organization, embedding structural constraints directly into functional representations. Imposing this population-level, physics-informed inductive bias can further enhance cross-subject and cross-dataset alignment. (3) Developing novel Temporal Adaptive Patch Embedding (TAPE) that enables scalable fMRI pretraining across heterogeneous TR values, overcoming key limitation

---

of existing models. (4) Introducing the first effective data augmentation for fMRI time series downsampling to hierarchical TR levels to accommodate heterogeneous TR distributions and enhance performance. (5) Finally, BrainHarmonix was benchmarked on a diverse set of downstream tasks, including the diagnosis of neurodevelopmental and neurodegenerative disorders, as well as the prediction of cognition. We demonstrate, for the first time, that complex brain morphology and dynamics can be deeply compressed into unified continuous-valued ID tokens that serve as holistic representations of the human brain.
2 Related Work
Recent brain foundation models have made significant advances in learning human brain representa- tions. BrainLM 4 and Brain-JEPA [5) pioneered self-supervised learning for fMRI time series using masked prediction and joint-embedding approaches, respectively. While these models demonstrated promising generalizability through global representations, they suffer from two critical shortcomings: (1) they ignore brain structural information, and (2) due to their standard choice of patch embedding layer in transformers, cannot accommodate heterogeneous TRs common across even within fMRI datasets. BDO [4l proposed a brain dynamics model based on stochastic optimal control, however, focuses exclusively on brain dynamics, similar to Brain-JEPA and BrainLM. In addition, BrainMass 7 has been proposed as the first foundation model for brain functional connectivity and pretrained on diverse IMRI datasets. However, it focuses exclusively on static functional connectivity without capturing brain structural information or temporal dynamics. On the other hand, BrainMVP 6 introduced self- cef-ssupervieed pretraining for 3D volumetric brain imaging that excels learning correspondence among multi-parametrie MRI, but fails to capture brain functional dynamics. This makes suboptimal for gaining comprehensive understanding of human brain functional organi- zation, capturing individual differences in behavior, and detecting abnormal alterations associated with neuropsychiatric disorders. To the best of our knowledge, Brain Harmony (BrainHarmoonix addresses these limitations the first multimodal foundation model that seamlessly integrates struc- tural morphology with functional dynamics while accommodating variable TR values. By unifying both modalities into ID tokens, BrainHarmonix creates compact and effective representational space that captures the holistic nature of the human brain.
3 Method
The pretraining of Brain Harmony (BrainHarmonix) comprises two sequential stages (Figure 2): (1) Unimodal Encoding (UE): we first separately train modality-peecifie encoders for Tl
(BrainHarmonix-S) and fMRI (BrainHarmonix--)) This separation allows flexible use of unpaired structural and functional data. For Braintarmonni-SS, we employ a 3D Masked Autoencoder (MAE)
15 that effectively captures structural information from the largest curation of T1 imaging datasets (given the widely adoption of MAE, readers are referred directly to Section 4.2 for inplementation details). For BrainHarmoniixFF we propose two significant innovations to masked brain modeling: first, a geometric harmonics- baused alignment method that pre-aligns brain dynamics with structural geometry; second, an innovative Temporal Adaptive Patch Embedding (TAPE) layer, enabling the encoder to flexibly accommodate any TR for the first time. Leveraging this unprecedented flexibility, we further introduce data augmentation techniques for fMRI time series, creating hierarchical TR values by downsampling high-resolution data. (2) Multimodal Fusion (MF): modality-specinfe representations are fused through set of learnable ID brain hub tokens. These tokens act as a representational bottleneck, explicitly trained to reconstruct both structural and functional latents, resulting in highly compact and unified latent space for human brain morphology and function.
3.1 Unimodal Encoding (UE)
In this subsection, we highlight two key innovations introduced in our approach for encoding fMRI dynamics (BrainHarmonix-F First, we propose geometric pre-alignment between brain dynamics and geometric harmonics, leveraging the foundational constraint that brain morphology inherently imposes on functional dynamics. Second, to effectively handle datasets with heterogeneous TRs, we introduce the Temporal Adaptive Patch Embedding (TAPE) layer, which enables token generation with consistent temporal length across varying TRs. For the encoding TI imaging (BrainHarmonnix- S), readers are referred to the implementation details provided in Section 4.2
3

---

A. Unimodal Encoding (UE)
B. Multimodal Fusion (MF)
Masking
Fusion on 69,360 T1-MMRI pairs
ES
D
Brain morphology (3D sMRI)
Pretrainsc 64, 504 volumes (-14 million images)
TAPE
Ea
(D Braân Hub
Observalion
F
D F
Ej P &
arw oonelnened by Harmor
TAPE
E
Target
F
Prediction
Brain dynamics (MRI time series)
T1 and (MRI latents
Discard during downstream adaplation
Pretrained from 70,933 runs
Figure 2: Pretraining of Brain Harmony (BrainHarmonix). A. Unimodal Encoding (UE): BrainHarmonix-S (Es) learns Tl structure via Masked Autoencoder (MAE); gray cubes rep- resent visible patches, while purple cubes are masked and reconstructed by the decoder (D's). BrainHarmonix-F (ε,) uses the Joint Embedding Predictive Architecture (JEPA) for fMRI, incor- porating our Temporal Adaptive Patch Embedding (TAPE) for heterogeneous TRs and geometric harmonics for cortical alignment, with the observation encoder (E';) and predictor (P) following standard JEPA. B. Multimodal Fusion (MF): The Harmonizer (H) fuses structural and functional latents into ID tokens (in green), then decoder (Ds D)) reconstruct modality speciffc latents.
3.1.1 Pre-alignment between brain dynamies and geometry
Recent neuroscientific research has revealed the profound relationship between brain phology and functional dynamics, demonstrat- ing that functional brain activity propagates as waves constrained by cortical geometry [8] In propose pre-align brain dynamics with morphology. Specifically, we posi- tion brain ROIs in the transformer using geomet- ric harmonics derived from population-leve corti- cal surface mesh, thereby enhancing the structure- function coherence of the learned latent space of brain dynamics.
M
BrainHarmonnix-F,
1/1
Spetial Bracuancy
Figure 3: Geometric harmonics.
Geometric harmonies are the natural, orthogonal vibration patterns of the brain's folded surface. Given mesh representation M of population-averaan cortical surface derived from T1 imaging, Laplace-Beltrami operator (LBO) AM constructed to capture local vertex-to-verte spatial relationships and cortical curvature. The corresponding eigenvalue problem can then be solved as follows:
AM vb = - v/, t
(I)
where v = (V1, u/.... is the sequence of geometric harmonics with the corresponding eigenval- ues (A1, )2........) ordered regarding spatial frequency (Figure 3). Each d E ERX is further downsampled through averaging within one ROI to formulate vi Nx1, where V represents the number of vertices in the mesh and N denotes the number of ROIS in a brain parcellation. First J downsampled harmonics v R xJ are selected for learning positional embedding. We incorporate a learnable linear layer to transform geometric harmonies V into positional embeddings F. e R where d denotes the embedding dimension of the transformer.

---

By explicitly encoding the geometric constraints into fMRI representations, we pre-align functional brain organization with cortical structure, enabling more effective integration during subsequent modality fusion. Moreover, embedding this physics-informed inductive bias, derived from population- level observations, can further enhance cross- subject and cross-dataset alignment.
3.1.2 Temporal Adaptive Patch Embedding (TAPE)
FMRT data collection spans scanners, sites and pulse-sequences that sample the BOLD signals anywhere from sub-second to several-second TRs. However, existing brain dynamics foun- dation models rely fixed TR for both pre- training and downstream tasks [., or downsam- ple datasets with higher temporal resolutions to match lower ones [5). This limitation restricts their ability to incorporate diverse datasets dur- ing pretraining and adapt flexibly to downstream tasks with varying TRs. Furthermore, downsam- pling inevitably sacrifices finer temporal details, reducing the richness of information encoded at higher temporal resolutions. The inability of existing models accommodate heterogeneous TRs stems from the use of a uniform patch size and a patch embedding layer with a fixed size in the transformer. When training on fMRI data with diverse TRs, employing a single patch size leads each patch to inadvertently represent different temporal durations across data. The embedding layer, however, fails to accurately interpret the varying temporal extents of the input. and well-defined temporal semantics, introducing ambiguity that hampers effective modeling of brain dynamics. This limitation ultimately degrades the quality of learned representations and impairs downstream performance, particularly in capturing brain-be associations.
T
T
Padding
TAPE
Attention Mask
Figure 4: Temporal Adaptive Patch Embedding (TAPE). Tokens represent the same temporal du- ration т.
Embedding weights are correspondingly resized. Shorter time series with fewer tokens zero-padded, with attention masks excluding padded tokens afterwards.
result, the tokens passed to the transformer lack consistent
To overcome this critical limitation, in BrainHarmonii-FF, we propose Temporal Adaptive Patch Embedding (TAPE) that dynamically accommodates varying TRs across fMRI data (Figure 4 We first define a consistent temporal duration т for any token and a base embedding weight wt E R corresponding the patch size k". Given an arbitrary IMRI time series with repetition time TR s, the corresponding patch size k and resized embedding weights w E R are computed as:
= round
: s
w = ((B) TT)
(2)
where w is obtained by pseudoinverse resize (PI-resize) [16, with the linear transformation matrix BK e R Since different fMRI scans may vary in total duration, patchifying with temporally consistent tokens could result in a varying number of tokens across time series. If the maximum number of tokens per time series across the dataset is m, any time series producing fewer tokens (n < m) will be zero-pat zero-padded followed by an attention mask, ensuring tokens derived from padding are excluded from attention computation (Figure
Previously, no established data augmentation techniques existed for fMRI time series; our TAPE uniquely supports arbitrary TRs, enabling the first-ever augmentation by downsampling high- resolution seans into diverse TRs, enhancing model performance (details in Section 4.1)
3.2 Multimodal Fusion (MF)
At BrainHarmonix's 's MF stage, we introduce learnable ID brain hub tokens that serve representa- tional bottleneck. These tokens are trained through the attention- model [17] to reconstruct both structural and functional latents, effectively capturing the shared information between modalities (Figure ID brain hub tokens foster a unified and compact latent space that encapsulates the holistic nature of brain morphology and dynamics.
Let Zg E Ne Zp E Nx x be the modalltity-speciie latents of one paired T1-MMR produced by BrainHarmonix and BrainHarmonix-E respectively, where dl is the common embedding dimen-
5

---

sion and Ns. NF are the numbers of tokens in each modality. We introduce a set of Nu learnable contiuuus-vvluued ID brain hub tokens Ho R xd shared by all pairs and optimized jointly with the network in MF. At every forward pass we concatenate the hubs with the two modality sequences and feed the resulting stream to the Harmonizer transformer (H):
Zo
Ho: Zs; ZF] E Ng* H H H()o)):NN E R RNN NNxx xd
where Zo is the concatenated input to and the hub tokens updated by H. Self-attention within H allows the 1D tokens to gather information from both structural and functional tokens, while also enabling cross-modal interactions between Z and Zr
Two lightweight decoders (Ds, DF) project H back into each modality's 's latent space (Figure B.).
Formally, our training in MF is defined as:
min 8x. 0Dg. PDF
Ltusion = IDs Zs 2 + [Dr - ZF
where 8z,, 0De dD, represents the parameters in H. Ds, and Dr, respectively, 11-113 denotes the Mean Square Error.
4 Experiments
4.1 Datasets
BrainHarmonix was pretrained on of the largest-scale neuroimaging datasets: UK Biobank (UKB) [18 19] and Adolescent Brain Cognitive Development (ABCD) [20]. From UKB, curated neuroimaging data of 43,1 12 participants aged between 44 and 83 years, comprising 46,455 Tl-weighted MRI scans and 40,162 resting-state IMRI time series (TR = 0.735 s). From ABCD, we included 11,221 participants (aged 8 to 11 years at baseline visit), consisting of 18,139 T1-weighted images and 30,771 resting-state IMRI time series (TR = 0.8 s).
Pre-training.
During the UE stage, total of 64,594 TI- images from both datasets were utilized for BrainHarmonix-S pretraining. For fMRI data augmentation, UKB data underwent temporal down- sampling by factors of to 3, resulting in TRs of 0.735 s, 1.47 2.205 s, and 2.94 ABCD data were downsampled by factors of 1 to 2, yielding TRs of 0.8 s, 1.6 s, and 2.4 s. Consequently. the total number of pretraining samples for Brainniimon was 252,961 (UKB: 40,162 x 4; ABCD: 30,771 x 3). In the MF stage, we extracted 69,360 matched T1-MMRI pairs from both datasets (one TI-weighted image could correspond data was parcellated into N = 400 ROIs with Schaefer-400 21 Further details regarding data preprocessing are provided in Appendix A
multiple IMRI runs within a single session). All IMRI
Downstream fine-tuning. We evaluated BrainHarmonix on six neuroimaging benchmark datasets. Three multi-site datasets focused neurodevelopmenntal disorder diagnosis (TR distributions detailed in Figure 7): Autism Brain Imaging Data Exchange datasets (ABIDE-I and ABIDE-II) for distinguishing Autism Spectrum Disorder (ASD) from controls, and the Attention Deficit Hyper- activity Disorder dataset (ADHD-200) for classifying ADHI) versus controls. On the other hand, three datasets assessed neurodegenerative disorders and cognitive function: Parkinson's Progression Markers Initiative (PPMI) (TR = 2.5s) for four-class classification involving controls, scans with- out evidence of dopaminergic deficit (SWEDD), prodromal cases, and Parkinson's 's disease (PD); Alzheimer's s Disease Neuroimaging Initiative (ADNI) (TR = 3.0s) for classification between controls and mild cognitive impairment (MCI); and the Lifespan Human Connectome Project Aging (HCP-A) dataset (TR = 0.8s) for predicting executive function (Flanker task scores). The results were averaged across three independent runs with distinct data splits traainvalidatiinn:esst 6:2:2). We adopted the data stratification approach in [22] for splitting the neurodevelopmental datasets. Detailed information regarding class distributions and preprocessing procedures for each benchmark dataset can be found in Appendix A
4.2 Implementation details
In UE, we adopted Vision Transformer-Base (ViT-B) 23 as backbone for BrainHarmonix-S and BrainHarmonix-F (Ey and εx). We employed MAE 15 as the pretraining framework for

---

Table 1: Comparison on neurodevelopmental disorder diagnosis. Results are averages over three random splits (standard deviations in Table 4.. The best results are highlighted in bold (* indicates statistical significance, p < 0.05), and second-bes results are underlined. Task details in Section 4.1
ABIDE-I
ABIDE-II
ADHD-200
Model
Morphology Dynamics Multi-TR
ACC%
F1%
ACC%
F1%
ACC%
F1%
Structure-based models
BrainMVP' 6
X
N.A.
56.50
62.46
55.71
62.16
67.72
43.97
BrainMVP 6
N.A.
55.06
64.43
55.63
58.76
62.59
49.95
BrainHarmoniix-S
X
N.A.
56.29
62.06
60.00
68.55
64.96
53.53
Function-bused models
BrainNetCNN 26
X
X
60.49
67.13
59.71
67.27
60.54
58.62
BrainGNN 27
X
56.72
65.71
58.71
66.48
62.24
60.67
BrainNerTF
22
X
X
56.73
64.75
62.03
67.64
61.91
62.68
BrainMass
X
65.64*
69.07
59.35
71.86
65.99
61.27
BrainLM 4
V
Brain- Prain-JEPAAI
X
BrainHarrmnnix-F
57.39
71.24
62.90
72.76
67.69
68.75
Multimodal model
BrainHarmonix
63.13
72.63*
66.67*
74.88*
70.09*
66.72
UniFormmr (8 as backbone:
PNETT3 29 as backbone.
BrainHarmonix-S with Brain-JEPA 5 for BrainHarmonix-FF In BrainHarmonix-FF it patchified fMRT time series into 1D patches, with the length dynamically determined by TR. Geometric harmonics and brain gradients (5) were each linearly projected, then averaged to produce the final positioning. TI images were randomly masked while we followed [5] to use spatiotemporal masking for fMRI.
In MF, harmonizer (H) was employed with ViT-B encoder, paired with MAE-style decoder (Ds & DE) whose design matches the encoder's size [15.. Throughout both MF and downstream fine-tuning, both BrainHammonix-S and BrainHarmonix-F were were frozen, providing modality latents only. For downstream fine-tuning, we peerage-pooled the brain hub tokens to generate global multimodal representation followed by linear projection head. The main results in Section 4.3 were all based on NB = 128 1D tokens. We employed FlashAttention [24] 25) in our self-attention implementation to improve computational efficiency and reduce memory usage. Each pre-training process utilized 8 NVIDIA H100 GPUs (80GB). The pretraining of H with 128 1D tokens took around 10 hours. The readers are refered to Appendix B for detailed optimization settings.
4.3 Main results
BrainHarmonix demonstrated strong generalization capabilities across neurodevelopmental and neurodegenerative disorder diagnoses as well as cognition prediction (Table I 2 As the first multimodal brain foundation model, BrainHarmonix was benchmarked against both structure-based and function-based neuroimaging models. For structure- 6), ttate-of-the-art structural foundation model originally designed for multi-parametri MRI. pretraining with T1 images only, we adopted BrainMVP's pretrained weights and fine-tune on downstream datasets. Many task-specific models for fMRI based on deep learning were proposed before foundation models emerged. These models could only be applied specific tasks rather than wide range of downstream applications [26] 27 22 30 BT. For functional comparisons, BrainHarmonix was evaluated against both task-specific (BrainNetCNN (6). BrainGNN (27]. and BrainNetTF (22 and foundational fMRI models (BrainMass M. BrainLM 4. and Brain-JEPA (D). Previous brain dynamics foundation models, including BrainLM and Brain-JEPA, are not able to handle heterogeneous TRs. Consequently, these models were only assessed on datasets with homogeneous TRs (PPMI, ADNI, and HCP-A) following their original downsampling strategies (Table 2). BrainMass was pretrained on our pretraining datasets following the original settings.
comparisons, we included BrainMVP
Given its incompatibility

---

Table 2: Comparison on neurodegenerative disease diagnosis and cognition prediction (standard
deviations in Table 5). Task details in Section 4.1
PPMI
ADNI
HCP-A
Model
Morphology Dynamics
Multi-TR
ACC%
F1%
ACC%
F1%
MAE
Structure-based models
BrainMVP'
x
N.A.
58.94
50.71
57.41
54.88
5.80
0.25
BrainMVP
x
N.A.
55.04
40.82
60.61
44.67
5.39*
0.36
BrainHHarmonix-S
/
X
N.A.
59.69
51.04
$7.59
56.09
6.05
0.38
Function-bused models
BrainNetCNN 26
X
56.59
46.59
56.57
54.59
0.23
BrainGNN 27
X
X
/
58.14
47.79
58.59
57.27
6.78
0.22
BrainNerTF
X
X
58.92
48.56
60.61
58.00
6.70
0.25
BrainMass 7
X
X
59.77
48.31
59.60
56.73
6.45
0.28
BrainLM
X
V
X
53.49
44.58
57.58
59.57
7.03
0.25
Brain-JEPA
X
V
X
60.36
48.76
59.60
60.78
5.62
0.26
BrainHarmonix--F
X
62.79
52.90
61.62
64.80
5.77
0.30
Multimodal model
BrainHarmonix
64.34
56.40*
64.65*
68.75*
6.56
0.42*
UniFormer (8 as backbone;
PUET33
29
backbone.
Scaling ABIDE-II
Scaling ADHD-200
69
72
Tuning (85.25M)
Fine Tuning
67
Linear Probing (0.0016M) 6M)
66.96
Linear Probing (0 0016M)
70.41
68
66.67
70.09
Accuracy (9)
65
Accuracy (%)
69.05
65.21
69
64
63
63
62.61
61.74
62.03
62
61.45
61.45
66.33
61
88
60
65
32
64
128
256
32
64
128
256
1D Token Numbers (Nw)
1D Token Numbers (Nr)
Figure 5: Scaling across different numbers of ID tokens in fine-tuning and linear probing. We plot both fine-tuning (85.25M trainable parameters) and linear probing (0.0015M trainable parameters). Increasing the token count from 32 to 256 steadily improves accuracy before reaching plateau. Notably, even our linear-probing approach achieves very strong performance, surpassing prior otate-of-the-art results despite using minimal set of learnable parameters.
Overall, BrainHarmonix consistently outperformed both structure-based and function-basse models. Among its ablations, BrainHarmonix-F, which exclusively captures functional dynamics, achieved superior performance compared to existing fMRI models, highlighting the effectiveness of modeling heterogeneous dynamics from large-scale neuroimaging data. On the other hand, BrainHammonix-S achieved performance that is superior or comparable BrainMVP through an MAE framework, pre- trained on large-scale T1 datasets without multi- cilttii-praetiiicc MRI. The performance of BrainHarmonix. can attributed to its pretraining on a significantly larger TI dataset, resulting in more robust brain morphological representations. The further improvements observed after cemecural-functionn fusion Barmonix underscore the significance of integrating multimodal heuristics for comprehensive human brain representation.

---

Ablation Study on ABIDE-II
Ablation Study on ADHD-200
with data augmentation
with data augmentation
data augmentation
66.67
70.09
Accuracy (%)
62.90
Accurecy (%)
66.67
67.69
12
60.29
81.16
62.32
66.33
67.69
64.63
58.
84.84
BrainHarmonlx wio pre-alignment
BrainHormonix-F
BrainHarmonix
BrainHarmonix wio pre-alignment
BrainHaronnix-
BrainHemmonxx
Figure 6: Ablation study showing the impact of pre-alignment, data augmentation, and mul- timodal fusion. In BrainHlarmonix-F without pre-alignment, sticked to the brain gradient positioning in Brain-JEPA, while the others (BrainHHrmmonxx-F and BrainHarmonix) were injected with geometric harmonics top of brain gradient (Section 4.2). Results were averaged over three random splits with error bars indicating standard deviations.
4.4 ID token scaling & linear probing
investigated how performance changed when we varied the number ID tokens from 32 to 256 and evaluated both fine-tuning and linear probing (Figure 5). As the number of tokens grows, model accuracy consistently increases highlighting the benefit richer token-based representations yet the gains begin to saturate from 128 256 tokens. Notably, our linear errebrgg approach, which uses only a simple linear head on top of the frozen BrainHarmonix, already achieves performance that ourperforms previous advanced baselines. This underscores the strength of the learned brain representations and their capacity to generalize even with minimal downstream adaptation.
Ablation study
We compared BrainHarmonix with its ablated versions in Figure 6 The comparison between the purple ("with data augmentation") and gray ("w/o data augmentation") lines demonstrates the consistent performance gains from augmenting fMRI time series through multi-TR downsampling. illustrates effectiveness of enriching the model's temporal representation. Comparing the center bars "BrainHarmonixx to the left ("BrainHarmonixxFF w/o pre-alignment") further reveals that pre-alignment of fMRI signals to cortical geometry significantly boosts performance. Finally, the rightmost bar ("BrainHarmonix" underscores the value of fusing structural and functional information: integrating these two complementary views of the brain yields the highest accuracy.
4.6 Latent space analyses and interpretation
Table 3: Comparison of significant modes t-SNE dimensions
Dimension
Model
Significant Modes
Avg. P-value
Avg. Correlation
Diml I-SNE
Brain-JEPA
7
0.00769 0.00456
0.1562
BrainHarmonix--
12
0.1717
Dim2 I-SNE
Brain-JEPA BrainHarmonix-F
8
0.0115
0.1506
15
0.00477
0.1726
number of; Avg. average; Dim dimension.
We extracted fMRI embeddings from BrainHarmonix- and Brain-JEPA (400 ROTs, each represented by 768-dimensional embedding) and applied t-SNE to project these embeddings onto a 2D plane. Specifically,, we correlated each dimension of the t-SNE embedding with each of the 200 geometric harmonic modes across 400 ROIs. Compared to Brain-JEPA, gcometry-constraiee embeddings

---

exhibit greater number of significantly correlated modes (p<0.05), with higher correlation strengths and significance levels on the top most significant modes (Table 3.. On the other hand, we applied the Fisher r-to-z transformation to all correlations from the 200 harmonics for each model and conducted a two-sample t-test. Results demonstrate that correlations from our model are significantly higher overall. The correlation strength and statistical significance from the top modes, along with the overall comparison, confirm that our model is constrained by structural information more than Brain-JEPA.
We examined the attention patterns between the 128 learned ID tokens and the Sadallty-speciffe tokens (400 fMRI ROI + 1200 TI tokens) in ASD diagnosis using ABIDE-II data. For the 400 fMRI ROI tokens, each is obtained by averaging all tokens within the corresponding ROI. We found differentiation in modality attention among the ID tokens: 93/128 tokens attended exclusively to fMRI, 30/128 exclusively to T1, and 5/128 tokens exhibited cross-modal attention. For cross-modal tokens, we found that they exhibited key structure-function coupling such as medial prefrontal cortex in brain morphometry and default model network in brain dynamics, which have previously been demonstrated in the literature to be associated with ASD.
For the 93 IMRI-specific tokens, further analysis revealed network-level functional differentiation relevant to ASD behavioral traits. Specifically, 60/93 were network network-specific, predominantly focusing on a single brain network (with >70% salient ROIs within one network), while the remaining 33 were identified as "bridge" tokens capturing interactions across multiple networks. Among the most salient network- sppcciffc tokens, temporoparietal network (implicated in social perception and language processing deficits), somatomotor network (associated with sensorimotor integration impairments), and default mode network (linked to mentalizing deficits) emerged prominently. The identified "bridge" tokens primarily captured interactions involving default, limbic, and control networks, reflecting impaired integration across sensorimotor, socioemotional, and higher-order cognitive processes mechanism implicated in pathophysiology of ASD.
5
Conclusion
In this paper, we introduced Brain Harmony (BrainHarmonix), the first multimodal brain founda- tion model that unifies structural morphology and functional dynamics into compact 1D tokens. By integrating geometric harmonics for structural-functional pre-alignment and introducing the Temporal Adaptive Patch Embedding (TAPE) for handling heterogeneous repetition times (TRs) in fMRI datasets, BrainHarmonix effectively bridges critical gaps existing in previous brain representation learning frameworks. Our approach provides unified, expressive latent space that significantly enhances the representation of complex brain morphology and dynamics. Extensive experiments demonstrated BrainHarmonix's superior generalization across diverse neuroimaging benchmarks, consistently outperforming state-of-the models in neurodevelopmental and neurodegeneratiiv disorder classification and cognition prediction. BrainHarmonix is positioned to fundamentally advance Al-driven neuroscience research and clinical applications through multimodal neuroimaging.
6 Limitations and Future Work
We acknowledge several limitations in our study and highlight directions for future work. First, although BrainHarmonix was pretrained on the largest curation of structural- datasets to date, the age distribution of the data could be further expanded to better represent the entire human lifespan. particularly infancy and young adulthood. On the other hand, jointly optimizing the unimodal encoders and the fusion module could potentially lead to further performance gains. Exploring efficient training strategies for such long-sequence transformer models, particularly in the context of neuroimaging, is a promising direction. Beyond the demonstrated gains in neuropsychiatric disease diagnosis and cognition prediction, our multimodal brain foundation model, BrainHarmonix, holds promise -if further developed- as an AI-driven brain digital twin: a neuroscientific tool capable of validating and potentially uncovering novel neuroscience insights such specific brain structure- function coupling related to human behavioral phenotypes. Realizing this potential, however, will require rigorous evaluation across diverse tasks and populations as well as systematic investigation into the model's interpretability and translational pathways.
neuroimaging
10

---

Acknowledgements
This study was supported by the Singapore National Medical Research Council NNMRC/OFLCG199Ma-0005. WMRC/CIRG//1485/2011 NMRC/CCAA-S//00022016 NMRC/MOH- 00707-01, NMRCICG/435 M009/207-NUHHNUHS, CIRG2 HLCCA23eeb-0004,
and OFIRG24Jul-0049), RIE2020 AME Programmatic Fund from A*STAR, Singapore (No. A20G8b0102), Ministry of Education IMOE-T2E441120-000 T2EP20220-0001), and Yong Loo Lin School of Medicine Research Core Funding, National University of Singapore, Singapore. We would like to acknowledge that GPU resource used in this work was supported by NUS IT's Research Computing group using grant numbers NUSREC-HPC-0001.
T2EP2-0023-00225, MOE-
References
[1] Nikos K Logothetis. What we can do and what we cannot with fmri Nature, 453(7197): 869-878, 2008.
[2] Russell A Poldrack and Martha J Farah. Progress and challenges in probing the human brain. Nature, 526(7573):371-379, 2015.
[3] Vince D Calhoun and Jing Sui. Multimodal fusion of brain imaging data: a key to finding the missing link (s) in complex mental illness. Biological psychiatry: cognitive neuroscience and neuroimaging, 1(3):230-22 244, 2016.
[4] Josue Ortega Caro, Antonio Henrique de Oliveira Fonseca, Syed A Rizvi, Matteo Rosati, Christopher Averill, James L Cross, Prateek Mittal, Emanuele Zappala, Rahul Madhav Dhodap- kar, Chadi Abdallah, et al. Brainlm: A foundation model for brain activity recordings. In The Twelfth International Conference on Leurning Representations.
[5] Zijian Dong, Ruilin Li, Yilei Wu, Thuan Tinh Nguyen, Joanna Chong, Fang Ji, Nathanael Tong, Christopher Chen, and Juan Helen Zhou. Brain-jepa: Brain dynamics foundation model with gradient positioning and spatiotemporal masking. Advances in Neural Information Processing Systems, 37:86048-86073 2024.
[6] Shaohao Rui, Lingzhi Chen, Zhenyu Tang, Lilong Wang. Mianxin Liu, Shaoting Zhang, and Xiaosong Wang. Brainmvp: Multi-modal vision pre-train for brain image analysis using parameri mri. arXiv preprint arXiv:2410.. 20040444000 2024.
[7] Yanwu Yang, Chenfei Ye, Guinan Su, Ziyao Zhang, Zhikai Chang, Hairui Chen, Piu Chan, Yue Yu, and Ting Ma. Brainmass: Advancing brain network analysis for diagnosis with large-scale self-supervised learning. IEEE Transactions on Medical Imaging, 2024.
[8] James C Pang, Kevin M Aquino, Marianne Oldehinkel, Peter A Robinson, Ben D Fulcher, Michael Breakspear, and Alex Fornito. Geometric constraints on human brain function. Nature, 618(7965),566-77, 2023.
[9] ADHD-200 consortium. The adhd-200 consortium: a model to advance the translational potential of neuroimaging in clinical neuroscience. Frontiers in systems neuroscience, 6:62, 2012.
[10] Adriana Di Martino, Chao-Gar Gan Yan, Qingyang Li, Erin Denio, Francisco X Castellanos, Kaat Alacrts, Jeffrey S Anderson, Michal Assaf, Susan Y Bookheimer, Mirella Dapretto, et al. The autism brain imaging data exchange: towards large-scale evaluation of the intrinsic brain architecture in autism. Molecular psychiatry, 19(6):6596667 2014.
[11] Adriana Di Martino, David O' connor, Bosi Chen, Kaat Alaerts, Jeffrey S Anderson, Michal Assaf, Joshua Balsters, Leslie Baxter, Anita Beggiato, Sylvie Bernaerts, al. Enhancing studies of the connectome in autism using the autism brain imaging data exchange ii. Scientific data, 4(1):1-15, 2017.
[12] Catie Chang and Gary H Glover. -ree-fryy dynamics of resting-state brain connectivity measured with fmri. Neuroimage, 10)188 2010.

---

[13] Vince D Calhoun, Robyn Miller, Godfrey Pearlson, and Tulay Adah. The chronnectome: time-varying connectivity networks as the next frontier in fmri data discovery. Neuron, 84(2): 262-274, 2014.
[14] Joonhyeong Park, Byoungwoo Park, Chang- -Be Bang, Jungwon Choi, Hyungjin Chung, Byung- Hoon Kim, and Juho Lee. A foundational brain dynamics model via stochastic optimal control. arXiv preprint arXiv:2502.0444 029922 2025.
[15] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[16] Lucas Beyer, Pavel Izmailov, Alexander Kolesnikov, Mathilde Caron, Simon Kornblith, Xiaohua Zhai, Matthias Minderer. Michael Tschannen, Ibrahim Alabdulmohsin, and Filip Pavetic. Flexivit: One model for all patch sizes. In Proceedings of the IEEE/CVE Conference on Computer Vision and Pattern Recognition, pages 14496-14506, 2023.
[17] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[18] Clare Bycroft, Colin Freeman, Desislava Petkova, Gavin Band, Lloyd T Elliott, Kevin Sharp, Allan Motyer, Damjan Vukcevic, Olivier Delaneau, Jared O'Connell, et al. The uk biobank resource with deep phenotyping and genomic data. Nature, 562(77266:203-209, 2018.
[19] Karla L. Miller, Fidel Alfaro-Almagro, Neal K Bangerter, David L. Thomas, Essa Yacoub, Junqian Xu, Andreas J Bartsch, Saad Jbabdi, Stamatios N Sotiropoulos, Jesper LR Andersson, al. Multimodal population brain imaging in the uk biobank prospective epidemiological study, Nature neuroscience, 19(11):1523-1566 2016.
[20] Betty Jo Casey, Tariq Cannonier, May I Conley, Alexandra O Cohen, Deanna M Barch, Mary M Heitzeg, Mary E Soules, Theresa Teslovich, Danielle Dellarco, Hugh Garavan, et al. The adolescent brain cognitive development (abed) study: imaging acquisition across 21 sites. Developmental cognitive neuroscience, 32:43-54, 2018.
[21] Alexander Schaefer, Ru Kong, Evan M Gordon, Timothy O Laumann, Xi-Nian Zuo, Avram J Holmes, Simon B Eickhoff, and BT Thomas Yeo. Local-global parcellation of the human cerebral cortex from intrinsic functional connectivity mri. Cerebral cortex, 28(9)33095-3114 114,, 2018.
[22] Xuan Kan, Wei Dai, Hejie Cui, Zilong Zhang, Ying Guo, and Carl Yang. Brain network transformer. Advances in Neural Information Processing Systems, 35:25586-25599, 2022.
[23] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. /929, 2020.
[24] Tri Dao, Daniel Y. Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficiee exact attention with 10-awareness. In Advances in Neural Information Processing Systems (NeurIPS), 2022.
[25] Tri Dao. FlashAttention-2: 2: Faster attention with better parallelism and work partitioning. In International Conference Learning Representations (ICLR), 2024.
[26] Jeremy Kawahara, Colin J Brown, Steven P Miller, Brian G Booth, Vann Chau, Ruth E Grunau, Jill G Zwicker, and Ghassan Hamarneh. Brainnetcnn: Convolutional neural networks for brain networks; towards predicting neurodevelopment. Neurolmage, 146:1038-1049 2017.
[27] Xiaoxiao Li, Yuan Zhou, Nicha Dvomek, Muhan Zhang, Siyuan Gao, Juntang Zhuang, Dustin Scheinost, Lawrence Staib, Pamela Ventola, and James S Dunean. Braingnn: Interpretable brain graph neural network for fmni analysis. Medical Image Analysis, 74:102233, 2021.
12

---

[28] Kunchang Li, Yali Wang, Junhao Zhang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unifying convolution and self-attention for visual recognition. IEEE Transactions on Puttern Analysis and Machine Intelligence, 45(10):12581-1266) 2023.
[29] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention-MICCA 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part 18, pages 234 241. Springer, 2015.
[30] Zijian Dong, Yilei Wu, Yu Xiao, Joanna Su Xian Chong, Yueming Jin, and Juan Helen Zhou. Beyond the snapshot: Brain tokenized graph transformer for longitudinal brain functional connectome embedding. In International Conference on Medical Image Computing and Computer-Asiisted Intervention, pages 348-357. Springer, 2023.
[31] Zijian Dong. Joanna Su Xian Chong, Bing Cai Kok, and Juan Helen Zhou. Coop-dhgnn: framework for joint classification and prediction of brain functional connectivity using sparse trajectory dataset with application to early dementia. In 2022 IEEE International Conference on Big Data (Big Data), pages 4972 4978. IEEE, 2022.
[32] Esten H Leonardsen, Han Peng, Tobias Kaufmann, Ingrid Agartz, Ole Andreassen, Elis- abeth Gulowsen Celius, Thomas Espeseth, Hanne F Harbo, Einar A Høgestøl, Ann-Marie De Lange, et al. Deep neural networks learn general and clinically relevant representations of the ageing brain. Neurolmage, 256:1 19210, 2022.
[33] Florent Ségonne, Anders M Dale, Evelina Busa, Maureen Glessner, David Salat, Horst Karl Hahn, and Bruce Fischl. A hybrid approach the skull stripping problem in mri. Neuroimage. 12(3):1000-1075, 2004.
[34] Mark Jenkinson, Christian F Beckmann, Timothy EJ Behrens, Mark W Woolrich, and Stephen M Smith. Fsl. Neuroimage, 62(2):782-790, 2012.
[35] Mark Jenkinson and Stephen Smith. A global optimisation method for robust affine registration of brain images. Medical image analysis, 5(2):143-156, 2001.
[36] Maria A Di Biase, Robert E Smith, Andrew Zalesky, Caio Seguin, et al. Connectomes for 40,000 uk biobank participants: A multi-modal, multi-scale brain network resource. Neurolmage, 283: 1 120407, 2023.
[37] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference Learning Representations, 2019. URL httpp:://ppenreview wtwpiiiwwwweeeforrree net/forum?
id-Bkg6Ricqy7
[38] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with joint- embedding predictive architecture. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15619-15629, 2023.
[39] Kenneth Marek, Danna Jennings, Shirley Lasch, Andrew Siderowf, Caroline Tanner, Tanya Simuni, Chris Coffey, Karl Kieburtz, Emily Flagg, Sohini Chowdhury, et al. The parkinson progression marker initiative (ppmi). Progress in neurobiology, 95(4):629-635 2011. 2011
[40] Jiaxing Xu, Yunhan Yang, David Huang, Sophi Shilpa Gururajapathy, Yiping Ke, Miao Qiao, Alan Wang, Haribalan Kumar, Josh McGeown, and Eryn Kwon. Data-driven network neu- roscience: On data collection and benchmark. Advances in Neural Information Processing Systems, 36:21841-21856 2023.
[41] Oscar Esteban, Christopher Markiewiez, Ross W Blair, Craig A Moodie, Ilkay Isik, Asier Erramuzpe, James D Kent, Mathias Goncalves, Elizabeth DuPre, Madeleine Snyder, et al. fmriprep: robust preprocessing pipeline for functional mri. Nature methods, 16(1):1111116 2019.
13

---

[42] Clifford R Jack Jr, Matt A Bernstein, Nick C Fox, Paul Thompson, Gene Alexander, Danielle Harvey, Bret Borowski, Paula Britson, Jennifer L. Whitwell, Chadwick Ward, et al. The alzheimer's s disease neuroimaging initiative (adni): Mri methods. Journal of Magnetic Reso- nance Imaging: An Official Journal of the International Society for Magnetic Resonance in Medicine, 27(4) 691, 2008
[43] Michael P Harms, Leah H. Somerville, Beau M Ances, Jesper Andersson, Deanna M Barch, Matteo Bastiani, Susan Y Bookheimer, Timothy B Brown, Randy L Buckner, Gregory C Burgess, et al. Extending the human connectome project across ages: Imaging protocols for the lifespan development and aging projects. Neuroimage, 183:972-984 984,, 2018.
[44] Jianxiao Wu, Jingwei Li, Simon B Eickhoff, Felix Hoffstaedter, Michael Hanke, BT Thomas Yeo, and Sarah Genon. Cross-cohort replicability and generalizability of connectivity-basee psychometric prediction patterns. Neuroimage, 262:119569, 2022.
[45] Joshua Faskowitz, Daniel Moyer, Daniel A Handwerker, Javier Gonzalez-Castillo Peter A Bandettini, Saad Jbabdi, and Richard Betzel. Commentary on pang et al. (203)) nature. bioRxiv, pages 2023-07, 2023.
[46] Zijian Dong, Yilei Wu, Zijiao Chen, Yichi Zhang, Yueming Jin, and Juan Helen Zhou. Prompt your brain: Scaffold prompt tuning for efficient adaptation of fmri pre-trained model. In International Conference Medical Image Computing and Computer-Assiste Intervention. pages 512-521. Springer, 2024.
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The last paragraph in introduction (Section D explicitly lists the paper's 's five
core contributions, All claims have been verified through experimental results in Section
and 4.5
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Limitations and future work are discussed Section 6
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
15

---

Answer: [NA]
Justification:
The paper does not include any theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: All datasets used in this work are publicly available, with detailed preprocessing information in Section [4.1 and Appendix [A] The model checkpoints and source code are included in the supplementary materials, accompanied by detailed instructions in the README file.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access a hosted model (e.g.. in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
16

---

5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Answer: [Yes]
Justification: All datasets used in this work publicly available. The model checkpoints and source code are included the supplementary materials, accompanied by detailed instructions in the README file.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips. cc/
for more details.
While we encourage the release of code and data, we understand that this might be possible, so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
//nips.cc/ hta.pss.cc/pbblccguuudeeee
for more details.
The authors should provide instructions data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Full codebase is provided in the supplementary materials. Detailed training settings with hyperparameters are provided Section 4.1 4.2 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Statistical significance is clearly labeled in the main results (Table I and with error bars in Table []
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
17

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
We have provided the computer resources in Section 4.2
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification: This work conforms the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Societal impacts are discussed Appendix C
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
18

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The proposed research poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
The original papers that produced the code package dataset are all properly cited.
Justification:
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
19

---

If assets are released, the license, copyright information, and terms of use in the thcode. has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, batasets.papersstt paperswi
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Code and model checkpoints provided in the supplementary materials, with detailed documentation.
Guidelines:
The answer NA means that the paper does not release assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The datasets of human subjects used this paper are all publicly available.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
The datasets of human subjects used in this paper are all publicly available.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
20

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM for what should or should not be described.
21

---

A Details of datasets
In this section, we detail the datasets used for pretraining and downstream evaluation of BrainHar- monix, describing their characteristics and associated preprocessing procedures.
ABIDE Distribution
ABIDE Distribution
ADHD200: Distribution
2,200
3,000
2,500
1.500
1857
0.913
2,000
1.000
3.000
2.000
1,60
1,000
Total Number of Butjects 708
Total Number of Subjects -
Total Number Subjects-
Figure 7: TR distributions three multi-site datasets.
A.1 T1 preprocessing (shared by all datasets)
All weeigttt images underwent a standardized preprocessing pipeline following 32: first, images were strippd using FreeSurfer [33]; then reoriented to match the standard orientation defined by FMRIB Software Library (FSL) [34]; and subsequently registered to the Montreal Neurological Institute (MNT) 152 template using FSL's linear registration tool (FLIRT) [35). Finally, images cropped to dimensions of 167 x 212 x 160 voxels, with voxel intensities normalized to a [0, 1] range.
A.2
Adolescent Brain Cognitive Development (ABCD)
The ABCD Study is the largest long-term study of brain development and child health in the United States [20]. We curated data from 11,221 participants, each with onc/two visits (baseline (aged 107- 133 months) and two-year follow-up), totaling 18.139 TI-weighted images and 30,771 resting-state [MRI time series (TR=0.8s). The IMRI preprocessing pipeline was as follows: fMRI were first aligned T1-weighted anatomical scans using boundary- based registration. Respiratory pseudo- motion artifacts were reduced by applying band-stop filter within the 0.31 1.31-0 Hz range. Frames exhibiting excessive motion- defined as framewise displacement (FD) greater than 0.3 mm or voxel- wise differentiated signal variance (DVARS) exceeding 50 were flagged. Each flagged frame, along with the preceding frame and two subsequent frames, was censored; additionally, any uncensored data segments containing fewer than five consecutive frames were also removed. Subsequently, nuisance signals- including global, white matter, ventricular signals, six head motion parameters, and their temporal derivatives were regressed out, with coefficients computed from uncensored data. Missing data from censored frames were interpolated using the Lomb-Scargle periodogram method, after which a band-pass filter of 0.009-0.08 Hz was applied. Finally, processed data were projected onto the FreeSurfer [33] fsaverage6 surface template and spatially smoothed using a Gaussian kernel with mm full-width at half-maxinne aalf-maximum (FWHM).
A.3
UK Biobank (UKB)
The UK Biobank is a large-scale biomedical database containing in-depth health information from UK
participants, with the neuroimaging component represents the largest brain imaging study [18 [9].
We curated neuroimaging data from 43,112 participants aged 44 to 83, comprising 46,455 T1-
weighted MRI and 40,162 fMRI time series (TR=0.735). Following Brain-JEPA (5), preprocessed fMRI data from [36].
we used the
A.4 Autism Brain Imaging Data Exchange (ABIDE-I and ABIDE-II)
The Autism Brain Imaging Data Exchange (ABIDE) is a multi-site, open-access initiative that aggregates structural and besting-state fMRI scans alongside rich phenotypic data from individuals with autism spectrum disorder (ASD) and matched typically developing controls to accelerate reproducible neuroimaging research. We curated neuroimaging data from two releases: ABIDE-1 10,, which contains 700 participants (320 control vs. 380 ASD) with paired TI and fMRI data collected across 20 different sites, and ABIDE-II (T]. which includes 553 participants (230 control
22

---

Table 4: Comparison on neurodevelopmental disorder diagnosis (standard deviations).
ABIDE-I
ABIDE-II
ADHID-200
Model
Morphology
Dynamics Multi-TR
ACC%
F1%
ACC%
F1%
ACC%
F1%
Structure-based models
BrainMVP
6
x
N.A.
0.85
1.68
4.44
13.11
10.57
23.24
BrainMVP
x
N.A.
8.20
5.85
2.95
18,85
1.18
2.71
BrainHarronii-SS
X
N.A.
6.32
5.74
3.79
4.03
1.56
4.36
Function-bused models
BrainNetCNN 26
X
X
/
1.38
0.75
1.33
1.90
0.59
3.70
BrainGNN 27
XXX
X
2.67
3,46
2.19
2.24
2.05
3.78
BrainNetTF
22
X
V
2.13
1.61
3.06
2.55
2.36
1.69
BrainMass
X
X
4.21
3.76
3.52
3.00
2.57
1.11
BrainI.M 4
X
V
X
Prain-JEPAAA
X
/
X
BrainHarmonix-F
X
V
V
1.38
2.30
1.32
1.06
4.60
2.77
Multimodal model
BrainHarmonix
4.31
1.30
2.18
1.02
4.57
3.31
UniFormmrr
28 as backbone; 2 UNET3D
29
as backbone.
Table 5: Comparison on neurodegeneratiie disease classification and cognition prediction (standard deviations).
PPMI
ADNI
HCP-A
Model
Morphology Dynamics Multi-TR
ACC%
F1%
ACC%
F1%
MAE
p
Structure-based models
BrainMVP'
6
X
N.A.
7.14
10.02
2.55
1.57
0.07
0.11
BrainMVP
6
X
N.A.
1.34
1.43
5.25
15.02
0.11
0.06
BrainHarmonix-S
X
N.A.
3.55
6.31
3.06
9.50
0.21
0.11
Function- based models
BrainNetCNN 26
XX
/
1.10
0.82
1.75
3.77
0.59
0.07
BrainGNN 27
/
0.00
0.45
4.63
8.32
0.63
0.06
BrainNetTF 22
1.35
1.06
3.03
2.30
0.23
0.01
BrainMass
1.63
3.28
1.75
5.81
0.69
0.06
BrainIM 4
X
2.33
1.56
5.25
5.08
0.26
0.03
Brain-JEPA
X
2.17
3.67
1.43
0.24
0.61
0.14
BrainHarmonix-P
/
V
2.33
3.65
1.75
1.91
0.73
0.11
Multimodal model
BrainHarmonix
3.55
6.31
4.63
3.87
0.56
0.12
UniFormer (28) as backbone;
2UNET3D 29 as backbone.
vs. 323 ASD) from 12 different sites. The distribution of heterogeneous. TR values is shown in Figure 7 The fMRI preprocessing begins by de-obliquing and reorienting each fMRI run, then discarding the initial volumes before applying slice-timing correction. Head motion was corrected with FSL's meflirt 34 and the time series were coregistered to each subject's T1 image using FreeSurfer's Next, nuisance signals including global, white matter, ventricular signals, six head motion parameters, and their temporal derivatives were regressed out and the data were despiked and band-pass filtered (0.009-0.08 Hz). Finally, the processed data were normalized to MNI space.
bbregister 33
23

---

Table 6: Pre-training settings.
config
value
Common configs
d
768 A
optimizer
37
optimizer momentum
B1, B2 = 0.9, 0.999
learning rate schedule
warmup cosine schedule
BruinHurmonix-S Harmonizer configs
start learning rate
learning rate
5 10
final learning
0
weight decay schedule
constant
weight decay
0.05
warmup epochs
40
BrainHarmonix-S patch size
16
Ng
1200
BrainHarrmonix-S total batch size
150 x GPU cards 1200
BrainHarmonix- training epochs
800
BrainHarmonix total batch size
15 8 GPU cards 120
BrainHarmonix training epochs
50
BrainHarmonix-F configs
200 8
patch size, k*
48
48 x 0.735 35.28 seconds
max number of tokens
18
Nr
400 x 18 = 7200
start learning rate
2.5 x 10
learning rate
5.7 x 10-5
final learning rate
1 10-6
weight decay schedule
cosine weight decay schedule 38
weight decay
0.05
final weight decay
0.4
EMA momentum schedule
linear 38
EMA start momentum
0.996
EMA final momentum
total batch size
64
8 GPU Cards 512
warmup epochs
10
training epochs
100
24

---

A.5 Attention Deficit Hyperactivity Disorder (ADHD-200)
The ADHD-200 dataset a multi-site, open-access repository of structural and resting-state fMRI scans with accompanying phenotypic measures from children and adolescents with attention- defici/hyperactivity disorder (ADHD) and matched typically developing controls. We curated neuroimaging data from 481 participants (292 control vs. 189 ADHD) with paired T1 and fMRI data collected across different sites. The distribution heterogeneous TR values is shown in Figure The ADHD-200 dataset underwent the same preprocessing procedure as the ABIDE datasets.
A.6 Parkinson's Progression Markers Initiative (PPMI)
The PPMI is longitudinal, multi-center, open-access dataset combining imaging, biospecimens, and detailed clinical assessments from Parkinson's disease (PD) patients, prodromal cohorts, and healthy controls accelerate biomarker discovery and risoase-progression research [39]. We utilized the open benchmark repository [40] preprocessed by IMRIPrep [4]), which contains data from 195 participants (15 control, 14 SWEDD, 53 Prodromal, and 113 PD patients).
A.7 Alzheimer's Disease Neuroimaging Initiative (ADNI)
The ADNI a longitudinal, multi-site study providing open-access neuroimaging, biomarker, genetic, and clinical data from cognitively normal (CN), mild cognitive impairment (MCI), and Alzheimer's disease (AD) participants to advance early diagnosis and therapeutic research [42]. We curated neuroimaging data from 164 participants (83 CN vs. 81 MCI) with paired T1 and fMRI data. The preprocessing procedure is the same as the ABIDE datasets.
A.8 Lifespan Human Connectome Project Aging (HCP-A)
The HICP-A dataset is Lifespan Human Connectome Project release that provides multimodal MRI and rich behavioral assessments from adults to elucidate brain connectivity changes across healthy aging 43] The resting-state fMRI data in MNI152 space underwent ICA-FIX denoising. We then performed nuisance regression to control for 24 motion parameters, white matter signal, CSF signal, and their temporal derivatives following (44).
B Additional implementation details
The default optimization settings for pretraining are detailed in Table We initialized all transformer blocks using the Xavier uniform method, as described in (15). For downstream adaptation, the default setting follows MAE [1], except for using AdmaW for linear probing.
C Additional analysis & discussion
C.1 Synthetic testing of TAPE
Table 7: Performance comparison on HCP-A test
MAE
Correlation
Original test set
6.56
0.42
Synthetic test set
6.69
0.39
To further evaluate TAPE's effectiveness, tested on both the original HCP-A test set and version with samples randomly downsampled by factors of and 2 (equal probability, leading to TR values 1.6, 2.4). The comparable performance across conditions demonstrates TAPE's robustness (Table 7)
C.2 Performance improvement: parameter count v.s. multimodal integration
We performed additional comparisons to better illustrate the performance gains from incorporating multiple modalities (2nd & 3rd columns in the Table g containing the results regarding accuracy
25

---

Table 8: Performance comparison across model sizes (accuracy%)
Single Modality (fMRI)
Concat
22M
86M
307M
ABIDE-II ADHD-200
62.90
63.19
64.06
66.67
66.95
67.69
68.36
69.39
70.09
70.40
(%)) and from introducing the harmonizer module for fusion (4th-6th columns). Specifically, we concatenated the embeddings from the frozen T1 and fMRI encoders and passed them through a trainable linear layer for the classification task. Furthermore, we conducted experiments using harmonizers of different sizes, ranging from 22M parameters to 307M parameters (4th-6th columns). The results above clearly demonstrate the performance improvements achieved both by adding modalities and by scaling the harmonizer module.
C.3
Ablation of ABCD dataset
Table 9: Model performance comparison on different training data (MCI classification on ADNI)
Model
ACC (%)
F1 Score (%)
Brain-JEPA using UKB only
59.60
60.78
BrainHarmonix-F using UKB only
60.67
63.34
BrainHarmonix-FF using both UKB ABCD
61.62
64.80
We conducted an ablation ADNI for MCI classification by pretraining BrainHarmonix F without ABCD data (Table D). We found still outperformed the original Brain-JEPA based on the same UKB dataset.
C.4 Ablation studies on ADNI
Table 10: Ablation on ADNI (accuracy%)
BrainHarmonix w/o pre-alignment
BrainHarmonix-P
BrainHarmonix
with DA
61.35
61.62
64.65
w/o DA
60.07
60.11
62.94
DA: data augmentation.
We additionally performed ablation studies ADNI (Table 10 (Table[IO regarding accuracy (%)), where observed a similar trend and performance pattern, reinforcing the effectiveness of our proposed model design.
C.5 Generalization to Asian clinical cohorts
We extended our evaluation to an Asian clinical cohort collected by Memory, Ageing and Cogni- tion Center (MACC), thereby assessing generalizability to non-Westemn populations and real- world clinical scenarios. Specifically, positive/negative participants, which holds significant climical value for AD prognosis and intervention. As shown in the Table II] BrainHarmont achieved ffte-of-the-aa art performance in this clinically relevant, in-house setting, underscoring its robustness and cross cooss-population generalizability.
performed additional task classification of amyloid-
C.6 Evaluation across data portions
We conducted additional analyses by scaling fine-tuning dataset using increasing proportions (20%, 40%, 60%, 80%, and 100%). The results regarding accuracy (%) are shown in the Table T2 Our results demonstrate clear and consistent scaling of performance with increasing data portions. Notably, compared with prior leading baseline BrainMass 59.35% ABIDE-II and 65.99% on
26

---

Table 11: Performance comparison on MACC
ACC (%)
F1 (%)
BrainMVP
65.83
53.64
BramnHarmonix-S
67.68
56.67
BrainNetCNN
57.57
52.00
BrainGNN
62.61
40.57
BrainNerTF
63.03
57.57
BrainMass
64.65
57.93
BrainLM
63.64
54.03
Brain-JEPA
66.67
59.18
Harmoniin
68.69
62.50
BrainHarmonix 74.75*
65.57
Table 12: Accuracy vs data portion (fine-tuning)
Portion (%)
20 %
40%
60%
80%
100%
ABIDE-II
Accuracy (%) 55.94
56.52
60.87
63.77
66.67
ADHD-200
Accuracy (%)
57.14
62.72
67.44
69.39
70.09
ADHD, BrainHarmonix achieves state-of-the-art performance even when fine-tuned on only 80% of the dataset, highlighting the efficiency and effectiveness of our pretrained representations.
On the other hand, we investigated the effect using different portions of the pretraining dataset. Specifically, applied identical sampling proportions both the UKB and ABCD datasets for pretraining. The corresponding results regarding accuracy (%) are reported in the Table [3] We observe that the model's performance improves the portion of the pretraining dataset increases.
C.7 Scaling with increasing token numbers
For completeness, we have included results with 512 and 1024 tokens as references in addtion to the results in the main content. shown in the Table 14] the accuracy (%) remains relatively stable beyond 256 tokens, confirming our initial observation.
C.8 Efficiency evaluation
included the pretraining time (on NVIDIA HI00 GPUs (80GB)), as well as finetuning time (on I H100 GPU) and inference time (on H100 GPU) on ABIDE-II, corresponding different model sizes (token counts) to provide a more comprehensive view of the computational cost the Table [5 Larger model or more token counts lead to longer computing time.
C.9 Discussion dynamic time warping (DTW)
Dynamic time warping (DTW) is an algorithm that measures the similarity between two temporal sequences, time series, that may vary in speed timing. It assumes meaningful temporal correspondence between sequences. However, in the context of resting-state fMRI, there is no ground truth temporal alignment across individuals, each subject's brain dynamics evolve independently and asynchronously. Therefore, applying DTW across different scans would impose artificial temporal correspondences not supported by the data.
C.10 Discussion on geometric harmonies in neuroimaging community
There are critiques to 8) which focus on the paper's ss claim that geometric harmonics, by themselves, can serve as "winner-take-all" solution for brain dynamics reconstruction, thereby diminishing the role of the structural connectome 45 However, the critiques do not affect the validity of our geometric pre-alignment. The harmonics in our work are only used to provide geometry-aware positioning, make claim that they can fully explain/reconstruel brain dynamics. On the other
27

---

Table 13: Accuracy vs data portion (pretraining)
Pretrain Portion
20% %
40 %
60%
80%
100%
ABIDE-I1 ADHD-200
59.12
62.90
64.35
65.21
66.67
64.96
65.64
67.01
68.37
70.09
Table 14: Scaling with increasing token numbers (accuracy%)
Dataset
Method
32
64
128
256
512
1024
ABIDE-II
Finetune 62.61
65.21
66.67
66.96
67.53
66.96
Linear Probe
61.45
61,45
61.74
62.03
62.32
62.89
ADHD-200
Finetune
67.69
69.05
70.09
70.41
70.41
70.75
Linear Probe
66.33
67.69
68.37
68.71
68.37
69.05
Table 15: Training and inference time comparison across model sizes (token numbers) on ABIDE-II
22M (128)
307M (128)
86M (32)
86M
86M (128)
86M (256)
86M (512)
86M (1024)
Pretraining Time
5h 10m
17h 9m
9h 20m
9h 26m
9h 37m
9h 45m
10h 23m
1hh 11m
Training Time
25m
41s
Inference Time
5.02s
7.19s
5.90s
5.89s
6,36s
5.77s
6.1
7.47
FT: fine tuning.
hand, the harmonics are averaged with large-scale functional gradients, so it does not have winner- take-all basis. Future work can explore how structural connectome and other biological principles can be encoded into the model and whether they can further improve brain representation learning and generalizability.
C.11 Potential failure mode
In our current evaluations, one notable case where BrainHarmonix underperforms is on the ADHD- 200. Its Fl slightly lower than BrainHarmonix-FF This likely due motion artifacts in T1, as ADHD patients exhibit increased head motion during MRI acquisition. Such motion introduces noise and negatively affects structural data quality, potentially reducing multimodal fusion performance. Future work will explore methods to improve robustness against data-qualic issues. On the other hand, although we have demonstrated data scaling effects, model performance under low-sample and few-shot learning scenarios remains an area for improvement. Future studies may address few-shot adaptation through approaches such as eaffciter-efficient fine-tuning or prompt-based tuning (46)
D Broader Impact
The integration and compression of multimodal neuroimaging signals could not only reduces storage and computational demands but also lays the groundwork for deployment on resource-constraingg platforms. Coupled with its capabilities for various downstream tasks, it may accelerates exploratory analyses, supports richer biomarker discovery, and drives improvements in diagnosis, prognosis, and personalized treatment planning. Moreover, its capacity to harmonize data across different scanners and acquisition protocols could enhance reproducibility and deepens our understanding of large-scale neuroimaging in both health and disease. However, these powerful capabilities also bring ethical responsibilities. Protecting patient confidentiality and ensuring data integrity are depental--eellommens must include rigorous de-identification procedures and secure data pipelines.
28


Reviews:
[{"review_id": "QhJeydj40t", "reviewer": "Reviewer_wvBG", "summary": "This paper proposes BrainHarmonix, a multimodal brain foundation model focused on joint encoding of structural morphology (structural MRI) and functional dynamics (fMRI). BrainHarmonix encodes structural information contained in T1-weighted structural MRI 3D volumes, allowing the model to respect inductive biases in the brain where morphological structure is thought to constrain functional dynamics. This addresses limitations in previous brain foundation models, which mainly focus on one modality of MRI data such as fMRI recordings. Inductive priors based on geometric harmonics are incorporated into the embeddings learned by the model, in addition to a novel temporal patch embedding framework for dealing with varying repetition times in different MRI datasets. Downstream results are presented on neurodevelopmental and neurodegenerative datasets, demonstrating that the individual components of BrainHarmonix show strong performance compared to other foundation models, and that the multimodal model further improves downstream performance. Ablation studies and scaling experiments exploring token counts and data augmentation are also presented.", "questions": "Do the authors have evidence that the latent space of brain representations learned by their model reflects structural constraints more than previous brain foundation models as a result of the incorporation of geometric harmonics?\nCan the authors clarify details of the TAPE encoding scheme, mainly (i) how the linear transformation matrices are learned and (ii) how much information loss there is (if any) when creating tokens across datasets with varying repetition time.\nCan the authors comment on end-to-end training versus modular training as described in their framework?\nHow much performance of the full multimodal model comes from the inclusion of multiple data modalities versus increased parameter count from having an additional architectural module (the harmonizer)?", "limitations": "yes", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\nMotivation & Clarity: The paper is well-written and well-motivated. The shortcomings of previous foundation models for brain activity which are limited in the modalities they encode are clearly explained, and the contributions that the authors are trying to make with this work are clear, mainly, a multimodal foundation model for brain representations that capture both structural and functional neuroimaging data with heterogeneous temporal resolution.\nContributions: The work makes clear contributions to address the two major shortcomings of previous foundation models: a temporal adaptive patch embedding scheme to account for variable time resolution across different MRI datasets, and added priors based on geometric harmonics for injecting structural information into the embeddings learned by the model.\nSignificance: This paper addresses a key consideration in the development of brain foundation models: learning generalizable representations for brain activity that account for both structural and functional, information. Furthermore, the work makes contributions to increase the architecture’s capability to be trained across multiple datasets despite heterogeneity in time resolution.\nDatasets and Experiments: The work uses large-scale, significant neuroimaging datasets for pretraining and evaluation, including the UK Biobank (UKB), Adolescent Brain Cognitive Development (ABCD), and multiple neurodevelopmental disorder and neurodegenerative datasets. The results shown on downstream datasets are impressive, and demonstrate the strength of the individual architecture components (BrainHarmonix-S and BrainHarmonix-F) while also demonstrating that for the most part, the multimodal model outperforms individual modality components and goes beyond single-modality performance.\nScaling & Ablation Studies: Scaling experiments are presented for the number of 1D tokens on neurodevelopmental disorder datasets, along with ablation studies \n\nWeaknesses\nLatent Space Analysis: A qualitative analysis of the embeddings learned by each unimodal encoder and/or the full multimodal model could add further results related to the embedding quality of the model. In particular, do the authors have evidence that the latent space of brain representations learned by their model reflects structural constraints more than previous brain foundation models as a result of the incorporation of geometric harmonics?\nClarity of Temporal Adaptive Patch Embedding (TAPE) method: A few details of the TAPE method are unclear from Section 3.1.2, for example: (i) is a linear transformation matrix B defined for every MRI dataset that is trained on, or rather per each repetition time s that occurs in pretraining datasets? And how are these matrices initialized/learned? (ii) The goal of creating tokens with consistent temporal duration is clear, however, how much guarantee is there that the linear transformation matrices create consistent tokens which are comparable across datasets? \nLack of End-to-End Training: Although the modular training framework allows each modality encoder to be trained independently before Multimodal Fusion (MF) training, the training framework does not allow a stage where all architecture components are trained end-to-end. This might hinder the model’s ability to create representations and encode all relevant information between both modalities, since the unimodal encoders are never updated together with the fusion architecture component. This may be difficult to address during a short rebuttal period, however discussion would be welcome regarding the training of individual components of the architecture: For example, why does the multimodal BrainHarmonix model underperform the individual performance of BrainHarmonix-F on the ADHD-200 dataset (in terms of F1-score), when in theory it has access to more information.\nParameter Count: Regarding the performance of BrainHarmonix versus individual modality components, additional discussion would be welcome on how much of the performance might stem from increased parameter count in the full BrainHarmonix model (with the Harmonizer model component) versus performance increase from having additional data modalities to perform downstream tasks.", "was_revised": true, "final_justification": "The authors addressed my concerns with clear explanations, additional analyses, and ablations, and while some limitations remain, the novelty, technical quality, and strong results justify a borderline accept.", "rebuttal": ">_Latent space analysis reflecting structural constraints_\n\nWe thank the reviewer for suggesting this valuable qualitative analysis. As recommended, we extracted fMRI embeddings from BrainHarmonix-F and Brain-JEPA (400 ROIs, each represented by a 768-dimensional embedding) and applied t-SNE to project these embeddings onto a 2D plane. _Since NeurIPS 2025 policy prohibits us from including new figures at this rebuttal stage, we performed a quantitative analysis comparing our model to Brain-JEPA_. Specifically, we correlated each dimension of the t-SNE embedding with each of the 200 geometric harmonic modes across 400 ROIs. Compared to Brain-JEPA, our geometry-constrained embeddings exhibit a greater number of significantly correlated modes (p<0.05), with higher correlation strengths and significance levels on the top 5 most significant modes (see table below). On the other hand, we applied the Fisher r-to-z transformation to all correlations from the 200 harmonics for each model and conducted a two-sample t-test. Results demonstrate that correlations from our model are significantly higher overall. The correlation strength and statistical significance from the top modes, along with the overall comparison, confirm that our model is constrained by structural information more than Brain-JEPA. We will incorporate the corresponding t-SNE visualization in the revised version.\n\n| Dimension        | Model      | # Significant Modes | Avg. P-value | Avg. Correlation |\n|:-----------------|:-----------|:-------------------:|:--------------------:|:-------------------------:|\n| Dim1 in t-SNE| Brain-JEPA |          7          |       0.00769        |          0.1562           |\n|                  | **Ours**   |       **12**        |      **0.00456**     |        **0.1717**         |\n| Dim2 in t-SNE| Brain-JEPA |          8          |       0.0115         |          0.1506           |\n|                  | **Ours**   |       **15**        |      **0.00477**     |        **0.1726**         |            |\n\n>_Clarification of TAPE_\n\nWe thank the reviewer for this insightful question, and we appreciate the opportunity to clarify our TAPE implementation:\n\nThe matrix $B^{k^{\\ast}}_k$ is defined for each TR. Specifically, for each TR ($s$) with corresponding patch size $k=round(\\tau/s)$, we construct $B^{k^{\\ast}}_k$ as a deterministic bilinear interpolation matrix that resizes embedding weights $\\omega^{\\ast}$ from the base size $k^{\\ast}$ to $k$. **This matrix is not learned** **but rather computed analytically** using standard bilinear interpolation. The learnable parameters here are $\\omega^{\\ast}$, which are optimized via backpropagation during pretraining. The $B^{k^{\\ast}}_k$ matrix provides a deterministic transformation to adapt $\\omega^*$ to different patch sizes corresponding to different TR. Implementation details can also be found in the code provided in the supplementary material.\n\nThe consistency of temporal duration is guaranteed both theoretically and empirically.\n\n**Theoretical guarantee:** $\\omega$ from the transformation of $\\omega^*$ with the pseudo-inverse of $B^{k^{\\ast}}_k$ matrix $(\\omega = ({B^{k^{\\ast}}_k}^T)^+\\omega^{\\ast})$ is the minimum-norm least-square solution that minimises $E_x[(<x, \\omega^{\\ast}> - <B^{k^{\\ast}}_kx, \\omega>)^2]$, where $<\\cdot,\\cdot>$ denotes inner product. A full derivation could be found in PI-resize work [1].\n\n**Empirical evidence:** To further evaluate TAPE’s effectiveness, we tested on both the original HCP-A test set and version with samples randomly downsampled by factors of 1 and 2 (equal probability, leading to TR values 1.6, 2.4). The comparable performance across conditions demonstrates TAPE's robustness.\n\n|                    | MAE  | Correlation |\n|:--------------------|:------:|:-------------:|\n| Original test set  | 6.56 | 0.42        |\n| Synthetic test set | 6.69 | 0.39        |\n\n>_Discussion on end-to-end training_\n\nWe thank the reviewer for the insightful comments regarding end-to-end training. The harmonizer is already pretrained with long input sequences (i.e., 7200 fMRI tokens + 1200 T1 tokens + the number of 1D tokens), resulting in considerable memory requirements during training. Enabling end-to-end training would substantially increase GPU memory consumption and training time, potentially taking more than two weeks per run under our current infrastructure, limiting our capacity of model development and comprehensive evaluation.\n\nHowever, we agree with the reviewer that jointly optimizing the unimodal encoders and the fusion module could potentially lead to further performance gains. Exploring efficient training strategies for such long-sequence transformer models, particularly in the context of neuroimaging, is a promising direction. We will discuss this important future work in the revised version.\n\nThe slightly lower F1 of BrainHarmonix compared to BrainHarmonix-F on ADHD-200 likely stems from motion artifacts in the T1 scans, especially prevalent among ADHD patients who tend to exhibit more head motion during MRI acquisition. Even though we have applied some quality control of T1, motion artifacts is unavoidable in T1 data of ADHD kids. This motion negatively impacts structural data quality, introducing noise that affects multimodal fusion. In contrast, fMRI data has gone through motion correction, of which the impact is less. We will investigate data-quality factors more systematically in future work.\n\n>_Performance improvement: parameter count vs. multimodal integration_\n\nWe thank the reviewer for the insightful question. We have added additional comparisons to better illustrate the performance gains from incorporating multiple modalities (2nd & 3rd columns in the table below containing the results regarding accuracy (%)) and from introducing the harmonizer module for fusion (4th-6th columns). Specifically, we concatenated the embeddings from the frozen T1 and fMRI encoders and passed them through a trainable linear layer for the classification task. Furthermore, we conducted experiments using harmonizers of different sizes, ranging from 22M parameters to 307M parameters (4th-6th columns). The results in the table below clearly demonstrate the performance improvements achieved both by adding modalities and by scaling the harmonizer module.\n\n|         | single modality (fMRI) | concat | 22M         | 86M   | 307M  |\n|---------|:-----------------:|:--------:|:-------------:|:-------:|:-------:|\n| AbideII | 62.90           | 63.19  | 64.06 | 66.67 | 66.95 |\n| ADHD    | 67.69           | 68.36  | 69.39 | 70.09 | 70.40 |::::::::::::\n\nReferences:\n\n[1] FlexiViT: One Model for All Patch Sizes. CVPR 2023."}, {"review_id": "XJAZi3IVf1", "reviewer": "Reviewer_nFU7", "summary": "The authors propose three modules to improve fMRI foundation models. First, they propose a multimodal fusion module that learns to fuse uni-modal representations into multi-modal tokens. Second, they show that pre-alignment to the geometric harmonics of the structural morphology of the brain improves performance on the ABIDE and ADHD-200 datasets. Third, the authors propose a patch embedding that can be used with multiple temporal sampling rates. This specific module also allows the authors to perform data augmentation during pre-training.", "questions": "The results in the original Brain-JEPA paper [1] for overlapping tasks is significantly lower in the authors' reproduction:\n- HCP-A Flanker: 0.406 (original) vs 0.26 (reproduction)\n- ADNI: 76.84 ACC, 86.32 F1 (original) vs 59.60 ACC, 60.78 F1 (reproduction)\nIs there a specific reason why the authors' fine-tuning reproductions differ so significantly in the performance reported in the original Brain-JEPA paper [1]? Could this be due to the frozen encoders? If so, can the authors include results that follow evaluation frameworks similar to previous work? I think this would significantly strengthen the work.\n\nCan the authors perform an ablation study to look at how the addition of the ABCD dataset affects model performance?\nI think this is relevant because the Brain-JEPA paper [1] the authors only used the UKBiobank dataset\n\nL198-199: Why do the authors sub-sample the UKB dataset 1-3 times, and the ABCD dataset 1-2 times?\n\nL228: Why are the unimodal encoders frozen while training the MF module? Have the authors ablated this and wouldn't steering the functional and structural encoders to learn multi-modal embeddings improve the representations, since this seems to be a central claim in the paper?\n\nIs there a specific reason that the authors chose to perform ablations in Figure 6 on the ABIDE and ADHD-200 datasets? Would it be more representative to use one neurodevelopment disorder and one neurodegenerative disorder dataset?\n\nThis is more of a personal observation, but it seems like the authors use a variety of visual styles in their figures (e.g. Figure 5 vs Figure 6 and Figure 1 vs Figure 3), is there any reason for this?\n\n[1] https://arxiv.org/pdf/2409.19407", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**\\\nThe authors compare their model across a wide range of datasets with many ablation studies, which makes me confident that the results are significant. Moreover, many of the modules the authors discuss are original and provide novel future directions in the field of foundational fMRI models. Especially the TAPE module and the ablation studies in Tables 1-3, and Figure 6 are great examples of the authors' original contributions and careful evaluations. \n\n**Major weaknesses**\\\nThe current manuscript mostly lacks clarity in terms of explaining the different modules that the authors use. Specifically, Section 3.1.2  is quite unclear, especially lines 155-165, but each of the sub-sections could use clarity improvements.\n\n**Minor weaknesses**\\\nL126L: How does the pre-alignment constrain the fMRI representations? I think I may be understanding this incorrectly, but aren't you using the downsampled harmonics in each ROI as a way to learn positional embeddings? If I understand it correctly, I think it is important to mention that Brain-JEPA [1] uses a similar way to encode positional embeddings, but using gradients instead of harmonics. Especially given the critiques/replies [2, 3] to the original work [4] that is the basis of the harmonics, it is important to ablate a few different ways to do learn positional embeddings. I am not arguing that these replies/critiques necessarily supersede the original work or the subsequent reply of the original authors [5], but I do think it is worth mentioning the critiques in the limitations section.\n\nTo further improve the paper, the authors could also compare their proposed model with the following work [6]. Since this paper was submitted to arxiv less than 6 months ago, however, I have not taken this into account when determining my scores.\n\n**Spelling and Grammer**\\\nL123: \"... will be selected for learning positional embedding.\" -> are selected to learn the positional embeddings from.\n\n[1] https://arxiv.org/pdf/2409.19407\n[2] https://www.biorxiv.org/content/10.1101/2023.07.20.549785v1.abstract\n[3] https://www.biorxiv.org/content/10.1101/2023.10.06.561240v1.abstract\n[4] https://www.nature.com/articles/s41586-023-06098-1\n[5] https://www.biorxiv.org/content/10.1101/2023.10.06.560797v1.abstract\n[6] https://arxiv.org/abs/2502.04892", "was_revised": true, "final_justification": "The authors were able to answer most of my concerns well, and I believe that this paper would be a good addition to Neurips 2025.", "rebuttal": ">_Clarification on TAPE_\n\nWe appreciate the opportunity to clarify Section 3.1.2 (lines 155-165).\n\nIn Figure 4, we illustrate the intuition behind TAPE. Given two fMRI time series with different TR ($s$), generating patches of a fixed temporal duration ($\\tau$) results in varying patch sizes $k$ (where $k = round(\\tau/s)$). Thus, a fixed-size embedding kernel ($\\omega$) is insufficient. Lines 155-165 describe how we transform a kernel of size $k^{\\ast}$ ($\\omega^{\\ast}$) to a kernel of an arbitrary length $k$ ($\\omega$) using the pseudo-inverse of the bilinear transformation matrix $B^{k^{\\ast}}_k$, as defined in Equation (2). $B^{k^{\\ast}}_k$ matrix is computed analytically using standard bilinear interpolation. The learnable parameters here are $\\omega^{\\ast}$, which are optimized via backpropagation during pretraining. The $B^{k^{\\ast}}_k$ matrix provides a deterministic transformation to adapt $\\omega^*$ to different patch sizes corresponding to different TR. Implementation details can also be found in the code provided in the supplementary material. \n\nTo further improve the clarity of lines 155-165, we will first quote Figure 4 to show the intuition, and add the definition of every symbol on first use at the beginning: “Let $\\tau$ be the temporal duration each patch should cover; $s$ the TR; $k$ an arbitrary patch length and $k^{\\ast}$ the base patch length”. Please refer to our response to Reviewer wvBG, \"Clarification of TAPE\" for the theoretical and empirical evidence supporting our TAPE method.\n\nWe appreciate the reviewer’s suggestion regarding clarity improvements. Besides Section 3.1.2, please let us know if any other section require further clarifications..\n\n>_Implementation and discussion of geometric harmonics_\n\nWe appreciate the reviewer’s insightful questions and pointer to the recent discussion on [1].\n\nWe appreciate the reviewer’s observation regarding our positional embedding strategy. While Brain-JEPA utilizes gradient-based positional embeddings, our approach extends this by integrating both brain geometric harmonics and gradients in a complementary manner. Specifically, as detailed in lines 224–226, we average both brain geometric harmonics and gradients after linear projection for positioning. This combination enables our model to capture both multi-scale oscillatory modes and large-scale gradient axes, promoting structure-function coherence in the latent space. It leads to better downstream performance compared to gradient only positioning in Brain-JEPA as evidenced in Figure 6. We will make these implementation differences clearer in the revision.\n\nThe critiques to [1] focus on the paper’s claim that geometric harmonics, by themselves, can serve as a “winner-take-all” solution for brain dynamics reconstruction, thereby diminishing the role of the structural connectome. However, the critiques do not affect the validity of our geometric pre-alignment. The harmonics in our work are only used to provide geometry-aware positioning, we make no claim that they can fully explain/reconstruct brain dynamics. On the other hand, the harmonics are averaged with large-scale functional gradients, so it does not have a winner-take-all basis. We will acknowledge these points in our revised version. Future work can explore how structural connectome and other biological principles can be encoded into the model and whether they can further improve brain representation learning and generalizability.\n\n>_Comparison to the arxiv work_\n\nWe appreciate the reviewer bringing this work [2] to our attention. However, the absence of publicly available code of this work currently prevents direct comparison. It proposes a single-modality foundation model focusing exclusively on brain dynamics, similar to Brain-JEPA and BrainLM, whereas our work unifies both structural morphology and functional dynamics into compact 1D tokens. We will acknowledge this recent work in the revised version.\n\n>_Grammer revision_\n\nWe thank the reviewer for pointing this out. We will revise the sentence accordingly.\n\n>_Evaluation of Brain-JEPA_\n\nWe appreciate the reviewer for observing this. The experiments were conducted using the official Brain-JEPA implementation for both pretraining and fine-tuning, during which the Brain-JEPA encoder was fully fine-tuned. The reported results were averaged over three random data splits. This is a more robust evaluation compared to the original results in the Brain-JEPA paper, which used one fixed data split. Therefore, the performance gap could come from the splitting difference. The three data split settings were applied consistently in all of our comparisons. We hope this clarifies the observed differences in performance and will explicitly state this in the revised version.\n\n>_Ablation of ABCD dataset_\n\nWe thank the reviewer for this valuable suggestion. It is important to highlight that previous methods, such as Brain-JEPA, cannot readily integrate datasets with heterogeneous TR values. In contrast, our proposed TAPE module allows datasets with heterogeneous TR, allowing us to add both UKB and ABCD into the pre-training to cover both the developing and aging brain. Following the reviewer’s recommendation, we conducted an ablation on ADNI for MCI classification by pretraining BrainHarmonix-F without ABCD data (see table below). We found it still outperformed the original Brain-JEPA based on the same UKB dataset.  We would also like to emphasize that the proposed data augmentation and pre-alignment techniques significantly contribute to the observed performance improvements as shown in Figure 6. We will include the ablation results in the revised version.\n\n| Model| ACC (%) | F1 Score (%) |\n|:--------------------------|:---------:|:---------------:|\n| Brain-JEPA using UKB only|59.60|60.78|\n| BrainHarmonix-F using UKB only |60.67|63.34|\n| BrainHarmonix-F using both UKB & ABCD|61.62| 64.80|\n\n>_Clarification on subsampling times_\n\nWe appreciate the reviewer’s question. Thanks to our proposed TAPE, our model uniquely handles heterogeneous TR, enabling us to introduce a novel data augmentation strategy for fMRI time series by downsampling. This approach enriches the dataset with a broader range of TR values and enlarges the sample size, significantly boosting model performance as demonstrated in Figure 6. Specifically, we downsample UKB data by factors ranging from 1 to 3, covering a TR range from ~0.7s to ~2.9s, and ABCD data by factors of 1 to 2, achieving a TR range up to 2.4s. Since single-band fMRI typically employs TR values between 2-3s and multi-band fMRI typically below 1s, further downsampling ABCD data would result in TR values exceeding 3s, which is uncommon in practical fMRI acquisitions and could lead to lower efficiency (longer training time).\n\n>_Frozen unimodal encoders_\n\nWe thank the reviewer for the insightful question. We chose to freeze the unimodal encoders during the fusion stage to allow the fusion module to flexibly integrate various pretrained unimodal representations. This design choice enables greater modularity and scalability, making it easier to adapt to different unimodal encoders trained under diverse settings without requiring full end-to-end retraining. As such, end-to-end training is not the main focus of this work. We would also like to emphasize that our current fusion module, even when trained with frozen unimodal encoders, already achieves significantly better performance than each unimodal model alone, demonstrating the effectiveness of our fusion strategy.\n\nMoreover, the harmonizer is already pretrained with long input sequences (i.e., 7200 fMRI tokens + 1200 T1 tokens + the number of 1D tokens), resulting in considerable memory requirements during training. Enabling end-to-end training would substantially increase GPU memory consumption and training time, potentially taking much longer time per run under our current infrastructure, limiting our capacity of model development and comprehensive evaluation.\n\nHowever, we agree with the reviewer that jointly optimizing the unimodal encoders and the fusion module could potentially lead to further performance gains. Exploring efficient training strategies for such long-sequence transformer models, particularly in the context of neuroimaging, is a promising direction and we will discuss it in future work in the revised version.\n\n>_More ablation studies_\n\nWe appreciate the reviewer’s thoughtful question. We initially chose ABIDE and ADHD-200 for our ablation studies because both datasets contain heterogeneous TR values. Following the reviewer’s suggestion, we additionally performed ablation studies on ADNI (see table below with results regarding accuracy (%)), where we observed a similar trend and performance pattern, reinforcing the effectiveness of our proposed model design. We will include this additional ablation in the revised version.\n\n|| BrainHarmonix w/o pre-alignment | BrainHarmonix-F | BrainHarmonix |\n|:------------------------:|:---------------------------------:|:-----------------:|:---------------:|\n| with data augmentation | 61.35| 61.62| 64.65|\n| w/o data augmentation  | 60.07| 60.11| 62.94|\n\n>_Figure styles_\n\nWe appreciate the reviewer’s observation regarding the style of our figures. Each figure is designed specifically to best convey the particular type of information presented. Figure 1 serves as a conceptual illustration, whereas Figure 3 visualizes the geometric harmonics generated from actual data. For Figure 5, we use circle sizes to clearly indicate the number of trainable parameters, a variable that is not applicable in Figure 6. However, we agree with the reviewer that a more consistent visual design would benefit readability, and we will refine the figures accordingly in the revised version.\n\nReferences:\n\n[1] Geometric constraints on human brain function. Nature 2023.\n\n[2] A Foundational Brain Dynamics Model via Stochastic Optimal Control. Arxiv 2025."}, {"review_id": "JfdgylqrXK", "reviewer": "Reviewer_Eo3b", "summary": "The paper introduces BrainHarmonix, a novel multimodal brain foundation model that unifies structural morphology from T1-weighted MRI and functional dynamics from fMRI into compact 1D brain-hub tokens. Unlike prior work that focused on either modality, BrainHarmonix bridges both through a two-stage pretraining pipeline: unimodal encoding (using MAE and JEPA architectures) followed by multimodal fusion using shared learnable tokens.\n\nKey contributions include:\n\nMultimodal Representation: First to integrate structural and functional data into shared 1D token space.\n\nGeometric Harmonization: Structural constraints (via geometric harmonics) are embedded into functional encoding, improving alignment.\n\nTemporal Adaptive Patch Embedding (TAPE): A strategy to handle heterogeneous TRs across fMRI datasets, a known limitation in prior models.\n\nEffective Data Augmentation: Downsampling-based augmentation for fMRI to enrich temporal dynamics.\n\nState-of-the-art Results: Demonstrates consistent improvements across diverse downstream tasks including ASD, ADHD, Parkinson’s, Alzheimer’s, and cognition prediction.\n\nThe model is trained on over 64,000 sMRI volumes and 70,000 fMRI scans, showing strong scalability and generalization.", "questions": "Why was dynamic time warping (DTW) not considered as a benchmark for aligning fMRI time series across TRs?\nDTW is a classic, domain-agnostic alignment strategy and could provide a useful contrast to the learned TAPE module. Including such a comparison, or at least discussing why it is inappropriate, would strengthen the argument for TAPE.\n\nHow interpretable are the 1D tokens?\nAre they anatomically localized, or do they represent global embeddings? Could the authors provide visualizations of token contributions to downstream tasks, or any biological interpretation?\n\nWhat are the limitations of using geometric harmonics across diverse populations or clinical scans?\nAre there risks of oversmoothing or biases introduced by relying on population-level cortical surfaces? How well does this generalize to atypical brains (e.g., patients with malformations)?\n\nCould the harmonizer be applied independently for zero-shot transfer across new datasets with only one modality?\nFor instance, can it be used to impute missing modalities or perform cross-modal translation? If not, could this be a direction for future work?", "limitations": "The authors acknowledge several limitations, such as restricted age coverage and the need for further generalization to other populations. However, a few areas are underdiscussed:\n\nDTW as a baseline: As noted, the omission of classical alignment techniques like DTW weakens the empirical justification for TAPE.\n\nInterpretability: More attention could be paid to understanding what the 1D tokens actually represent, beyond task performance.\n\nSynthetic testing of TRs: Take a dataset with a fixed TR, resample it with different TRs. Now compare original data with artificially and irregularly sampled TR to test out TAPE?\n\nGeneralization risks: While datasets are large, they mostly reflect Western populations; generalization to non-Western cohorts, clinical settings, or non-standard acquisition protocols is untested.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 2, "significance": 4, "originality": 4, "strengths_and_weaknesses": "Strengths\nThe unification of morphology and dynamics into 1D tokens is a significant conceptual step forward. Prior models typically focus on only one modality. Authors address a clear need in neuroimaging—handling heterogeneous TRs and fusing modalities.\n\nTAPE and geometric harmonics-based alignment are novel and well-motivated, offering practical solutions to known limitations.\nEvaluated across several challenging datasets, with detailed ablations (Figure 6) and scaling experiments (Figure 5).\n\nThe submission includes detailed implementation and preprocessing steps, openly available data, and code release in the supplementary.\n\nWeaknesses\nLack of Baseline Against Dynamic Time Warping: The proposed TAPE method addresses variable TRs, but omits comparisons to standard time-alignment methods like dynamic time warping (DTW), which would offer a useful baseline.\n\nInterpretability: While the compact tokenization is elegant, interpretability of these 1D tokens—especially with respect to individual behavioral or clinical traits—is underexplored.\n\nFusion Mechanism: The harmonizer is somewhat abstract; more insights into how the attention mechanism learns cross-modal representations (e.g., attention maps, token influence) would enhance clarity.\n\nEvaluation Scope: While six benchmarks are substantial, generalization to clinical settings outside large public datasets (e.g., low-resource or single-site studies) is not explored.", "was_revised": false, "final_justification": "", "rebuttal": ">*Lack of Baseline Against Dynamic Time Warping*\n\nWe appreciate the reviewer’s valuable suggestion. However, we believe that DTW is not a suitable baseline in this setting. DTW assumes a meaningful temporal correspondence between sequences. However, in the context of resting-state fMRI, there is no ground truth temporal alignment across individuals, as each subject’s brain dynamics evolve independently and asynchronously. Therefore, applying DTW across different scans would impose artificial temporal correspondences not supported by the data.\n\nWe thank the reviewer for bringing this method to our attention. We will add the above discussion in the related work and TAPE section.\n\n>*Interpretability of 1D tokens and attention maps*\n\nWe appreciate the reviewer’s insightful comments regarding the interpretability of our 1D tokens. *Although NeurIPS 2025 policy prohibits us from including new figures at this rebuttal stage, we would like to briefly summarize key insights from our new analyses*.\n\nWe examined the attention patterns between the 128 learned 1D tokens and the modality-specific tokens (400 fMRI ROI  + 1200 T1 tokens) in ASD diagnosis using ABIDE-II data. For the 400 fMRI ROI tokens, each is obtained by averaging all tokens within the corresponding ROI. We found differentiation in modality attention among the 1D tokens: 93/128 tokens attended exclusively to fMRI, 30/128 exclusively to T1, and 5/128 tokens exhibited cross-modal attention. For cross-modal tokens, we found that they exhibited key structure-function coupling such as medial prefrontal cortex in brain morphometry and default model network in brain dynamics, which have previously been demonstrated in the literature to be associated with ASD.\n\nFor the 93 fMRI-specific tokens, further analysis revealed network-level functional differentiation relevant to ASD behavioral traits. Specifically, 60/93 were network-specific, predominantly focusing on a single brain network (with >70% salient ROIs within one network), while the remaining 33 were identified as “bridge” tokens capturing interactions across multiple networks. Among the most salient network-specific tokens, temporoparietal network (implicated in social perception and language processing deficits), somatomotor network (associated with sensorimotor integration impairments), and default mode network (linked to mentalizing deficits) emerged prominently. The identified “bridge” tokens primarily captured interactions involving default, limbic, and control networks, reflecting impaired integration across sensorimotor, socioemotional, and higher-order cognitive processes - a mechanism implicated in pathophysiology of ASD. Please refer to our response to Reviewer wvBG, “Latent space analysis reflecting structural constraints” for the interpretation of the geometry-constrained fMRI latent space.\n\nWe will incorporate corresponding visualization into our revised manuscript.\n\n>*Limited generalization exploration beyond public Western datasets*\n\nWe thank the reviewer for raising this important point regarding generalization. To address this, we extended our evaluation to an Asian clinical cohort collected from a university hospital memory clinic (single site), thereby assessing generalizability to non-Western populations and in real-world clinical scenarios. Specifically, we performed an additional task not included in the current version - classification of amyloid-positive/negative cognitively normal participants, which holds significant clinical value for AD prognosis and intervention. As shown in the table below, BrainHarmonix achieved state-of-the-art performance in this clinically relevant, in-house setting, underscoring its robustness and cross-population generalizability. We will incorporate these additional results in the revised version.\n\n|                 | ACC (%)    | F1 (%)     |\n|:-----------------|:--------|:--------|\n| BrainMVP        | 65.83  | 53.64  |\n| BrainHarmonix-S | 67.68  | 56.67  |\n| BrainNetCNN     | 57.57  | 52.00  |\n| BrainGNN        | 62.61  | 40.57  |\n| BrainNetTF      | 63.03  | 57.57  |\n| BrainMass       | 64.65  | 57.93  |\n| BrainLM         | 63.64  | 54.03  |\n| Brain-JEPA      | 66.67  | 59.18  |\n| BrainHarmonix-F | 68.69  | 62.50  |\n| **BrainHarmonix**   | **74.75*** | **65.57*** |\n\n>*Population-level geometric harmonics*\n\nWe thank the reviewer for highlighting this important consideration. Using geometric harmonics derived from a group-level cortical surface provides robust, stable alignment by capturing common cortical geometry across individuals, thereby facilitating generalization and comparability across diverse datasets. In contrast, subject-specific cortical surfaces can be highly variable and prone to errors. However, we agree that combining group-level and individualized geometric harmonics may further enhance sensitivity and robustness, especially for atypical populations (e.g. patients) or the developing/aging brain. We will explore individualized (or age-specific) geometric harmonics in future studies which could further enhance model performance for clinical applications.\n\n>*Zero-shot cross-modal transfer with harmonizer*\n\nWe thank the reviewer for this insightful suggestion. While the current harmonizer requires paired multimodal inputs, a promising future direction would be to adapt it for zero-shot transfer and cross-modal imputation. One possible solution is to train modality-specific adapters, which could learn mappings from single-modality inputs to multimodal latent representations, enabling the harmonizer to impute missing modalities or perform cross-modal translation in scenarios where only one modality is available. We will add this in future work.\n\n>*Synthetic testing of TRs*\n\nWe appreciate this insightful suggestion. To further evaluate TAPE’s effectiveness, we tested on both the original HCP-A test set and version with samples randomly downsampled by factors of 1and 2 (equal probability, leading to TR values 1.6, 2.4). The comparable performance across conditions demonstrates TAPE's robustness.\n\n|                    | MAE  | Correlation |\n|:--------------------|:------:|:-------------:|\n| Original test set  | 6.56 | 0.42        |\n| Synthetic test set | 6.69 | 0.39        |"}, {"review_id": "gpcOqnaCd7", "reviewer": "Reviewer_746W", "summary": "This paper presents BrainHarmonix, a multimodal brain foundation model that unifies structural (T1-weighted MRI) and functional (fMRI) neuroimaging data into compact 1D token representations. The proposed method comprises a two-stage modular training framework: unimodal encoding (UE) followed by multimodal fusion (MF) via shared brain hub tokens. UE includes two key modules: (a) Geometric Harmonics-Based Pre-alignment of fMRI dynamics with cortical structure to impose structural constraints on functional encoding, and (b) Temporal Adaptive Patch Embedding (TAPE) to handle varying fMRI repetition times (TRs), addressing a major barrier in prior models. Experimental results demonstrate BrainHarmonix's strong performance across neurodevelopmental and neurodegenerative disorder diagnosis and cognitive prediction benchmarks. Ablation studies highlight the critical roles of multimodal fusion, pre-alignment, and TR-aware augmentation. The model shows scalable performance and efficient generalization even with linear probing.", "questions": "Please refer to the items listed under the Weaknesses section.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The paper is well-organized and accessible. Each section and complex concepts  are explained with visual aids and equations.\n- The work addresses critical gaps in the brain foundation model literature, including the lack of multimodal integration and the inability to generalize across fMRI datasets with variable TRs.\n\nWeaknesses:\n- The paper claims: \"We demonstrate, for the first time, that complex brain morphology and dynamics can be deeply compressed into unified continuous-valued 1D tokens that serve as holistic representations of the human brain.\" However, prior foundation models have also aimed to learn unified representations to capture complex brain morphology and dynamics. To clarify the novelty of this work, it would be helpful to distinguish it more clearly from existing approaches. A comparative table analyzing similarities and differences with previous models would strengthen the contribution and help position this work in the broader context of neuroimaging foundation models.\n- Despite the large pretraining datasets, the data is primarily sourced from the UK Biobank and ABCD, which focus on specific age groups (middle-aged adults and children, respectively). As acknowledged by the authors, early life (infancy) and young adulthood are underrepresented. Moreover, there is limited evidence of geographical, scanner, or demographic diversity, which may affect generalization.\n- To further demonstrate the method’s effectiveness, it would be helpful to present results on additional datasets and tasks, such as the Human Connectome Project for Early Psychosis (HCP-EP) and the Transdiagnostic Connectome Project (TCP).\n- The method relies on downstream fine-tuning after pretraining. To better showcase its effectiveness, it would be helpful to include a few-shot setting—for example, using an increasing portion of data (e.g., 5%, 10%, ...100%) from a neuroimaging benchmark dataset for fine-tuning.\n- Figure 5 presents results on scaling across different numbers of 1D tokens (32, 64, 128, 256) during fine-tuning and linear probing. The curves show that increasing the token count from 32 to 256 steadily improves accuracy. However, a clear performance plateau is not observed, contrary to what is suggested in the text. To make this claim more convincing, it would be helpful to include results with additional token counts (e.g. 512, 1024) to show whether performance eventually plateaus, continues increasing, or begins to decrease at some point.\n- It would be helpful to include training and inference times under varying experimental settings—for example, different token counts. Further, scalability experiments assessing how model performance evolves with increasing model complexity or data availability would strengthen the analysis. Such an evaluation would clarify the effects of scaling and illuminate the trade-offs in runtime and memory usage.\n- The paper could benefit from a deeper exploration of where BrainHarmonix underperforms, particularly in fine-grained cognitive prediction tasks or low-sample regimes. Including an analysis of failure modes would strengthen the paper.", "was_revised": true, "final_justification": "Most of concerns are well-addressed.", "rebuttal": ">*Distinction from prior foundation models*\n\nWe appreciate the reviewer’s feedback and would like to clarify this important point. To the best of our knowledge, we are indeed the first to deeply compress both brain morphology and dynamics into unified representations via 1D tokens as a multimodal brain foundation model. Regarding “prior foundation models”, we would greatly appreciate it if the reviewer could point out specific works we might have overlooked.\n\nAs acknowledged by the other three reviewers, our proposed BrainHarmonix is a significant conceptual step forward, addressing a clear need in neuroimaging by effectively handling heterogeneous TRs and multimodal data fusion, thus providing novel directions for future research in brain foundational models.\n\nMoreover, Table 1 and 2 in our paper provide how the proposed model differs from previous brain foundation models. For further clarity, we summarize the key distinctions below:\n\n|Model| Modality | Compact 1D latent representation of brain | Capture brain morphology | Capture brain dynamics | Handle heterogeneous TR | Constrained by brain cortical surface |\n|:------------|:-------------------------------------------------------|:-------------------------------------------:|:--------------------------:|:------------------------:|:-------------------------:|:---------------------------------------:|\n| BrainMVP   | Structural MRI (volumetric data)| No | Yes| No | N.A.| N.A.|\n| BrainMass  | fMRI (functional connectivity)| No| No| No| Yes                     | No|\n| BrainLM    | fMRI (time series) | No | No| Yes | No | No|\n| Brain-JEPA | fMRI (time series)  | No| No | Yes| No| No|\n| **BrainHarmonix**       | Structural MRI (volumetric data) + fMRI (time series) | Yes  | Yes| Yes| Yes| Yes|\n\nWe hope this comparison clarifies the unique contributions of our work.\n\n>*Data diversity*\n\nWe thank the reviewer’s insightful comment and would like to emphasize that we have explicitly acknowledged the limitations regarding age coverage in our pretraining datasets within the \"limitations and future work\" section.\n\nImportantly, our current pretraining data already extends the lifespan coverage compared to prior state-of-the-art brain dynamics models such as BrainLM and Brain-JEPA, expanding from middle-aged and older adults to children as well, thereby enabling downstream analyses of neurodevelopmental disorders that previous models did not address. Moreover, our pretraining datasets inherently include scanner diversity, for example, ABCD data was acquired from multiple vendors (Siemens, GE, Philips). To further demonstrate generalizability across different geographical and demographic contexts, we have extended our downstream evaluations to include an Asian clinical cohort collected from a university hospital memory clinic, effectively assessing our model in real-world clinical scenarios. Please refer to our response to Reviewer Eo3b, \"Limited generalization exploration beyond public Western datasets\" for the detailed results.\n\n>*Evaluation on additional datasets and tasks*\n\nWe thank the reviewer for this valuable suggestion. Due to the limited rebuttal time and resources, we apologize that we are unable to complete the downloading, preprocessing, and evaluation of our model on the suggested datasets. Nevertheless, we extended our evaluation to an Asian clinical cohort collected from a university hospital memory clinic, thereby assessing generalizability to non-Western populations and in real-world clinical scenarios, as correctly pointed out by the reviewer.\n\nSpecifically, we performed an additional task not included in the current version - classification of amyloid-positive/negative cognitively normal participants, which holds significant clinical value for AD prognosis and early intervention. Please refer to our response to Reviewer Eo3b, \"Limited generalization exploration beyond public Western datasets\" for detailed results. We will incorporate the suggested HCP-EP and TCP for downstream evaluation in our future work.\n\n>*Fine-tuning evaluation across data portions*\n\nWe appreciate the reviewer’s suggestion regarding few-shot evaluation. To address this, we conducted additional analyses by scaling the fine-tuning dataset using increasing proportions (20%, 40%, 60%, 80%, and 100%). The results regarding accuracy (%) are shown in the table below. Our results demonstrate a clear and consistent scaling of performance with increasing data portions. Notably, compared with prior leading baseline BrainMass 59.35% on AbideII and 65.99% on ADHD, BrainHarmonix achieves state-of-the-art performance even when fine-tuned on only 80% of the dataset, highlighting the efficiency and effectiveness of our pretrained representations. We will include this analysis in the revised version.\n\n|   | Portion (%) | 20%   | 40%   | 60%   | 80%   | 100%  |\n|:-----------|:-------------|:-------|:-------|:-------|:-------|:-------|\n| ABIDE-II  | Accuracy (%)| 55.94 | 56.52 | 60.87 | 63.77 | 66.67 |\n| ADHD-200  | Accuracy (%)| 57.14 | 62.72 | 67.44 | 69.39 | 70.09 |\n\n>*Scaling behavior with increasing token numbers*\n\nWe appreciate your valuable comments. In our preliminary experiments, we observed that the performance tends to stabilize when the number of tokens reaches around 256. For completeness, we have included results with 512 and 1024 tokens as references. As shown in the table below, the accuracy (%) remains relatively stable beyond 256 tokens, confirming our initial observation.\n\n| Dataset| Method| 32    | 64    | 128   | 256   | 512   | 1024  |\n|:--------------|:--------------|:-------|:-------|:-------|:-------|:-------|:--------|\n| AbideII  | Finetune     | 62.61 | 65.21 | 66.67 | 66.96 | 67.53 | 66.96 |\n|              | Linear Probe | 61.45 | 61.45 | 61.74 | 62.03 | 62.32 | 62.89 |\n| ADHD     | Finetune     | 67.69 | 69.05 | 70.09 | 70.41 | 70.41 | 70.75 |\n|              | Linear Probe | 66.33 | 67.69 | 68.37 | 68.71 | 68.37 | 69.05 |\n\n>*Scalability and efficiency evaluation*\n\nWe thank the reviewer for the valuable suggestions. We have added scalability experiments by varying the size of the harmonizer model from 22M parameters to 307M parameters. The results regarding accuracy (%) in the table below show performance improvement from 22M to 86M, while the gain from 86M to 307M is only marginal.\n\n|         | 22M | 86M| 307M  |\n|:---------|:-------------:|:-------:|:-------:|\n| AbideII | 64.06 | 66.67 | 66.95 |\n| ADHD    | 69.39 | 70.09 | 70.40 |\n\nOn the other hand, we investigated the effect of using different portions of the pretraining dataset. Specifically, we applied identical sampling proportions to both the UKB and ABCD datasets for pretraining. The corresponding results regarding accuracy (%) are reported in the table below. We observe that the model’s performance improves as the portion of the pretraining dataset increases.\n\n| Pretrain Portion | 20% | 40% | 60% | 80%| 100%  |\n|:------------------|:------------:|:------------:|:------------:|:------------:|:-------:|\n| AbideII| 59.12| 62.90| 64.35| 65.21      | 66.67 |\n| ADHD| 64.96| 65.64 | 67.01| 68.37 | 70.09 |\n\nFollowing the reviewer’s suggestion, we have also included the pretraining time (on 8 NVIDIA H100 GPUs (80GB)), as well as finetuning time (on 1 H100 GPU) and inference time (on 1 H100 GPU) on AbideII, corresponding to different model sizes (token counts) to provide a more comprehensive view of the computational cost in the table below. Larger model or more token counts lead to longer computing time. \n\n|        | 22M (128)    | 307M (128)    | 86M (32)     |86M (64)       | 86M (128)     | 86M (256)     | 86M (512)      | 86M (1024)   |\n|----------------|--------------|--------------|---------------|---------------|---------------|---------------|---------------|---------------|\n| Pretrain Time  | 5h 10m   | 17h 9m       | 9h 20m        | 9h 26m        | 9h 37m        | 9h 45m        | 10h 23m       | 11h 11m       |\n| FT Training Time  | 0h 21m 52s   | 1h 07m 28s   | 0h 25m 33s    | 0h 26m 54s    | 0h 27m 41s    | 0h 29m 54s    | 0h 30m 17s    | 0h 31m 54s    |\n| Inference Time | 5.02s | 7.19s        | 5.90s         | 5.89s         | 6.36s         | 5.77s         | 6.11s         | 7.47s         |\n\nOverall, our further analysis highlights a trade-off: larger models or increased token counts generally enhance performance but result in longer computation times.\n\n>*Failure mode and limitation analysis*\n\nWe appreciate the reviewer’s suggestion to analyze potential failure modes and areas of underperformance. In our current evaluations, one notable case where BrainHarmonix underperforms is on the ADHD-200. Its F1 is slightly lower than BrainHarmonix-F.  This is likely due to motion artifacts in T1, as ADHD patients exhibit increased head motion during MRI acquisition. Such motion introduces noise and negatively affects structural data quality, potentially reducing multimodal fusion performance. Future work will explore methods to improve robustness against data-quality issues. On the other hand, although we have demonstrated data scaling effects in the above discussion, model performance under low-sample and few-shot learning scenarios remains an area for improvement. Future studies may address few-shot adaptation through approaches such as parameter-efficient fine-tuning or prompt-based tuning."}]

Author rebuttal:

Meta-review: This paper proposes a multimodal foundation model for brain imaging data that unifies structural and functional fMRI into a 1D continuous token representations. The method overcomes shortcomings of prior works by allowing for variable temporal resolution and incorporating structural priors. Their foundation model is trained on a large scale dataset of sMRI and fMRI data, demonstrating convincing strengths in downstream performance and scaling. There were some questions regarding the TAPE method -- mostly wrt to clarification -- but these were well addressed. This paper addresses an important problem in multimodal and heterogeneous data in brain imaging, and has answered some important practical questions with convincing results, and I believe this will have value towards NI applications such as diagnosis.

I therefore recommend the paper for acceptance as a poster.
Decision: Accept (poster)