Title: Brain Harmony: A Multimodal Foundation Model Unifying Morphology and Function into 1D Tokens
Venue: NeurIPS 2025
Accepted: yes
Keywords: Multimodal Neuroimaging; Brain Foundation Model; 1D Tokens; Representation Learning; Neuropsychiatric Disorders; Cognition
Abstract: We present **Brain Harmony  (BrainHarmonix)**, the first multimodal brain foundation model that unifies structural morphology and functional dynamics into compact 1D token representations. The model was pretrained on two of the largest neuroimaging datasets to date, encompassing 64,594 T1-weighted structural MRI 3D volumes (～ 14 million images) and 70,933 functional MRI (fMRI) time series. BrainHarmonix is grounded in two foundational neuroscience principles: *structure complements function* - structural and functional modalities offer distinct yet synergistic insights into brain organization; *function follows structure* - brain functional dynamics are shaped by cortical morphology. The modular pretraining process involves single-modality training with geometric pre-alignment followed by modality fusion through shared brain hub tokens. Notably, our dynamics encoder uniquely handles fMRI time series with heterogeneous repetition times (TRs), addressing a major limitation in existing models. BrainHarmonix is also the first to deeply compress high-dimensional neuroimaging signals into unified, continuous 1D tokens, forming a compact latent space of the human brain. BrainHarmonix achieves strong generalization across diverse downstream tasks, including neurodevelopmental and neurodegenerative disorder classification and cognition prediction - consistently outperforming previous approaches. Our models - pretrained on 8 H100 GPUs - aim to catalyze a new era of AI-driven neuroscience powered by large-scale multimodal neuroimaging.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents BrainHarmonix, a multimodal brain foundation model that unifies structural (T1-weighted MRI) and functional (fMRI) neuroimaging data into compact 1D token representations. The proposed method comprises a two-stage modular training framework: unimodal encoding (UE) followed by multimodal fusion (MF) via shared brain hub tokens. UE includes two key modules: (a) Geometric Harmonics-Based Pre-alignment of fMRI dynamics with cortical structure to impose structural constraints on functional encoding, and (b) Temporal Adaptive Patch Embedding (TAPE) to handle varying fMRI repetition times (TRs), addressing a major barrier in prior models. Experimental results demonstrate BrainHarmonix's strong performance across neurodevelopmental and neurodegenerative disorder diagnosis and cognitive prediction benchmarks. Ablation studies highlight the critical roles of multimodal fusion, pre-alignment, and TR-aware augmentation. The model shows scalable performance and efficient generalization even with linear probing.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The paper is well-organized and accessible. Each section and complex concepts  are explained with visual aids and equations.\n- The work addresses critical gaps in the brain foundation model literature, including the lack of multimodal integration and the inability to generalize across fMRI datasets with variable TRs.\n\nWeaknesses:\n- The paper claims: \"We demonstrate, for the first time, that complex brain morphology and dynamics can be deeply compressed into unified continuous-valued 1D tokens that serve as holistic representations of the human brain.\" However, prior foundation models have also aimed to learn unified representations to capture complex brain morphology and dynamics. To clarify the novelty of this work, it would be helpful to distinguish it more clearly from existing approaches. A comparative table analyzing similarities and differences with previous models would strengthen the contribution and help position this work in the broader context of neuroimaging foundation models.\n- Despite the large pretraining datasets, the data is primarily sourced from the UK Biobank and ABCD, which focus on specific age groups (middle-aged adults and children, respectively). As acknowledged by the authors, early life (infancy) and young adulthood are underrepresented. Moreover, there is limited evidence of geographical, scanner, or demographic diversity, which may affect generalization.\n- To further demonstrate the method’s effectiveness, it would be helpful to present results on additional datasets and tasks, such as the Human Connectome Project for Early Psychosis (HCP-EP) and the Transdiagnostic Connectome Project (TCP).\n- The method relies on downstream fine-tuning after pretraining. To better showcase its effectiveness, it would be helpful to include a few-shot setting—for example, using an increasing portion of data (e.g., 5%, 10%, ...100%) from a neuroimaging benchmark dataset for fine-tuning.\n- Figure 5 presents results on scaling across different numbers of 1D tokens (32, 64, 128, 256) during fine-tuning and linear probing. The curves show that increasing the token count from 32 to 256 steadily improves accuracy. However, a clear performance plateau is not observed, contrary to what is suggested in the text. To make this claim more convincing, it would be helpful to include results with additional token counts (e.g. 512, 1024) to show whether performance eventually plateaus, continues increasing, or begins to decrease at some point.\n- It would be helpful to include training and inference times under varying experimental settings—for example, different token counts. Further, scalability experiments assessing how model performance evolves with increasing model complexity or data availability would strengthen the analysis. Such an evaluation would clarify the effects of scaling and illuminate the trade-offs in runtime and memory usage.\n- The paper could benefit from a deeper exploration of where BrainHarmonix underperforms, particularly in fine-grained cognitive prediction tasks or low-sample regimes. Including an analysis of failure modes would strengthen the paper.\n\nQUESTIONS:\nPlease refer to the items listed under the Weaknesses section.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces BrainHarmonix, a novel multimodal brain foundation model that unifies structural morphology from T1-weighted MRI and functional dynamics from fMRI into compact 1D brain-hub tokens. Unlike prior work that focused on either modality, BrainHarmonix bridges both through a two-stage pretraining pipeline: unimodal encoding (using MAE and JEPA architectures) followed by multimodal fusion using shared learnable tokens.\n\nKey contributions include:\n\nMultimodal Representation: First to integrate structural and functional data into shared 1D token space.\n\nGeometric Harmonization: Structural constraints (via geometric harmonics) are embedded into functional encoding, improving alignment.\n\nTemporal Adaptive Patch Embedding (TAPE): A strategy to handle heterogeneous TRs across fMRI datasets, a known limitation in prior models.\n\nEffective Data Augmentation: Downsampling-based augmentation for fMRI to enrich temporal dynamics.\n\nState-of-the-art Results: Demonstrates consistent improvements across diverse downstream tasks including ASD, ADHD, Parkinson’s, Alzheimer’s, and cognition prediction.\n\nThe model is trained on over 64,000 sMRI volumes and 70,000 fMRI scans, showing strong scalability and generalization.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\nThe unification of morphology and dynamics into 1D tokens is a significant conceptual step forward. Prior models typically focus on only one modality. Authors address a clear need in neuroimaging—handling heterogeneous TRs and fusing modalities.\n\nTAPE and geometric harmonics-based alignment are novel and well-motivated, offering practical solutions to known limitations.\nEvaluated across several challenging datasets, with detailed ablations (Figure 6) and scaling experiments (Figure 5).\n\nThe submission includes detailed implementation and preprocessing steps, openly available data, and code release in the supplementary.\n\nWeaknesses\nLack of Baseline Against Dynamic Time Warping: The proposed TAPE method addresses variable TRs, but omits comparisons to standard time-alignment methods like dynamic time warping (DTW), which would offer a useful baseline.\n\nInterpretability: While the compact tokenization is elegant, interpretability of these 1D tokens—especially with respect to individual behavioral or clinical traits—is underexplored.\n\nFusion Mechanism: The harmonizer is somewhat abstract; more insights into how the attention mechanism learns cross-modal representations (e.g., attention maps, token influence) would enhance clarity.\n\nEvaluation Scope: While six benchmarks are substantial, generalization to clinical settings outside large public datasets (e.g., low-resource or single-site studies) is not explored.\n\nQUESTIONS:\nWhy was dynamic time warping (DTW) not considered as a benchmark for aligning fMRI time series across TRs?\nDTW is a classic, domain-agnostic alignment strategy and could provide a useful contrast to the learned TAPE module. Including such a comparison, or at least discussing why it is inappropriate, would strengthen the argument for TAPE.\n\nHow interpretable are the 1D tokens?\nAre they anatomically localized, or do they represent global embeddings? Could the authors provide visualizations of token contributions to downstream tasks, or any biological interpretation?\n\nWhat are the limitations of using geometric harmonics across diverse populations or clinical scans?\nAre there risks of oversmoothing or biases introduced by relying on population-level cortical surfaces? How well does this generalize to atypical brains (e.g., patients with malformations)?\n\nCould the harmonizer be applied independently for zero-shot transfer across new datasets with only one modality?\nFor instance, can it be used to impute missing modalities or perform cross-modal translation? If not, could this be a direction for future work?\n\nLIMITATIONS:\nThe authors acknowledge several limitations, such as restricted age coverage and the need for further generalization to other populations. However, a few areas are underdiscussed:\n\nDTW as a baseline: As noted, the omission of classical alignment techniques like DTW weakens the empirical justification for TAPE.\n\nInterpretability: More attention could be paid to understanding what the 1D tokens actually represent, beyond task performance.\n\nSynthetic testing of TRs: Take a dataset with a fixed TR, resample it with different TRs. Now compare original data with artificially and irregularly sampled TR to test out TAPE?\n\nGeneralization risks: While datasets are large, they mostly reflect Western populations; generalization to non-Western cohorts, clinical settings, or non-standard acquisition protocols is untested.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose three modules to improve fMRI foundation models. First, they propose a multimodal fusion module that learns to fuse uni-modal representations into multi-modal tokens. Second, they show that pre-alignment to the geometric harmonics of the structural morphology of the brain improves performance on the ABIDE and ADHD-200 datasets. Third, the authors propose a patch embedding that can be used with multiple temporal sampling rates. This specific module also allows the authors to perform data augmentation during pre-training.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\\\nThe authors compare their model across a wide range of datasets with many ablation studies, which makes me confident that the results are significant. Moreover, many of the modules the authors discuss are original and provide novel future directions in the field of foundational fMRI models. Especially the TAPE module and the ablation studies in Tables 1-3, and Figure 6 are great examples of the authors' original contributions and careful evaluations. \n\n**Major weaknesses**\\\nThe current manuscript mostly lacks clarity in terms of explaining the different modules that the authors use. Specifically, Section 3.1.2  is quite unclear, especially lines 155-165, but each of the sub-sections could use clarity improvements.\n\n**Minor weaknesses**\\\nL126L: How does the pre-alignment constrain the fMRI representations? I think I may be understanding this incorrectly, but aren't you using the downsampled harmonics in each ROI as a way to learn positional embeddings? If I understand it correctly, I think it is important to mention that Brain-JEPA [1] uses a similar way to encode positional embeddings, but using gradients instead of harmonics. Especially given the critiques/replies [2, 3] to the original work [4] that is the basis of the harmonics, it is important to ablate a few different ways to do learn positional embeddings. I am not arguing that these replies/critiques necessarily supersede the original work or the subsequent reply of the original authors [5], but I do think it is worth mentioning the critiques in the limitations section.\n\nTo further improve the paper, the authors could also compare their proposed model with the following work [6]. Since this paper was submitted to arxiv less than 6 months ago, however, I have not taken this into account when determining my scores.\n\n**Spelling and Grammer**\\\nL123: \"... will be selected for learning positional embedding.\" -> are selected to learn the positional embeddings from.\n\n[1] https://arxiv.org/pdf/2409.19407\n[2] https://www.biorxiv.org/content/10.1101/2023.07.20.549785v1.abstract\n[3] https://www.biorxiv.org/content/10.1101/2023.10.06.561240v1.abstract\n[4] https://www.nature.com/articles/s41586-023-06098-1\n[5] https://www.biorxiv.org/content/10.1101/2023.10.06.560797v1.abstract\n[6] https://arxiv.org/abs/2502.04892\n\nQUESTIONS:\nThe results in the original Brain-JEPA paper [1] for overlapping tasks is significantly lower in the authors' reproduction:\n- HCP-A Flanker: 0.406 (original) vs 0.26 (reproduction)\n- ADNI: 76.84 ACC, 86.32 F1 (original) vs 59.60 ACC, 60.78 F1 (reproduction)\nIs there a specific reason why the authors' fine-tuning reproductions differ so significantly in the performance reported in the original Brain-JEPA paper [1]? Could this be due to the frozen encoders? If so, can the authors include results that follow evaluation frameworks similar to previous work? I think this would significantly strengthen the work.\n\nCan the authors perform an ablation study to look at how the addition of the ABCD dataset affects model performance?\nI think this is relevant because the Brain-JEPA paper [1] the authors only used the UKBiobank dataset\n\nL198-199: Why do the authors sub-sample the UKB dataset 1-3 times, and the ABCD dataset 1-2 times?\n\nL228: Why are the unimodal encoders frozen while training the MF module? Have the authors ablated this and wouldn't steering the functional and structural encoders to learn multi-modal embeddings improve the representations, since this seems to be a central claim in the paper?\n\nIs there a specific reason that the authors chose to perform ablations in Figure 6 on the ABIDE and ADHD-200 datasets? Would it be more representative to use one neurodevelopment disorder and one neurodegenerative disorder dataset?\n\nThis is more of a personal observation, but it seems like the authors use a variety of visual styles in their figures (e.g. Figure 5 vs Figure 6 and Figure 1 vs Figure 3), is there any reason for this?\n\n[1] https://arxiv.org/pdf/2409.19407\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes BrainHarmonix, a multimodal brain foundation model focused on joint encoding of structural morphology (structural MRI) and functional dynamics (fMRI). BrainHarmonix encodes structural information contained in T1-weighted structural MRI 3D volumes, allowing the model to respect inductive biases in the brain where morphological structure is thought to constrain functional dynamics. This addresses limitations in previous brain foundation models, which mainly focus on one modality of MRI data such as fMRI recordings. Inductive priors based on geometric harmonics are incorporated into the embeddings learned by the model, in addition to a novel temporal patch embedding framework for dealing with varying repetition times in different MRI datasets. Downstream results are presented on neurodevelopmental and neurodegenerative datasets, demonstrating that the individual components of BrainHarmonix show strong performance compared to other foundation models, and that the multimodal model further improves downstream performance. Ablation studies and scaling experiments exploring token counts and data augmentation are also presented.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\nMotivation & Clarity: The paper is well-written and well-motivated. The shortcomings of previous foundation models for brain activity which are limited in the modalities they encode are clearly explained, and the contributions that the authors are trying to make with this work are clear, mainly, a multimodal foundation model for brain representations that capture both structural and functional neuroimaging data with heterogeneous temporal resolution.\nContributions: The work makes clear contributions to address the two major shortcomings of previous foundation models: a temporal adaptive patch embedding scheme to account for variable time resolution across different MRI datasets, and added priors based on geometric harmonics for injecting structural information into the embeddings learned by the model.\nSignificance: This paper addresses a key consideration in the development of brain foundation models: learning generalizable representations for brain activity that account for both structural and functional, information. Furthermore, the work makes contributions to increase the architecture’s capability to be trained across multiple datasets despite heterogeneity in time resolution.\nDatasets and Experiments: The work uses large-scale, significant neuroimaging datasets for pretraining and evaluation, including the UK Biobank (UKB), Adolescent Brain Cognitive Development (ABCD), and multiple neurodevelopmental disorder and neurodegenerative datasets. The results shown on downstream datasets are impressive, and demonstrate the strength of the individual architecture components (BrainHarmonix-S and BrainHarmonix-F) while also demonstrating that for the most part, the multimodal model outperforms individual modality components and goes beyond single-modality performance.\nScaling & Ablation Studies: Scaling experiments are presented for the number of 1D tokens on neurodevelopmental disorder datasets, along with ablation studies \n\nWeaknesses\nLatent Space Analysis: A qualitative analysis of the embeddings learned by each unimodal encoder and/or the full multimodal model could add further results related to the embedding quality of the model. In particular, do the authors have evidence that the latent space of brain representations learned by their model reflects structural constraints more than previous brain foundation models as a result of the incorporation of geometric harmonics?\nClarity of Temporal Adaptive Patch Embedding (TAPE) method: A few details of the TAPE method are unclear from Section 3.1.2, for example: (i) is a linear transformation matrix B defined for every MRI dataset that is trained on, or rather per each repetition time s that occurs in pretraining datasets? And how are these matrices initialized/learned? (ii) The goal of creating tokens with consistent temporal duration is clear, however, how much guarantee is there that the linear transformation matrices create consistent tokens which are comparable across datasets? \nLack of End-to-End Training: Although the modular training framework allows each modality encoder to be trained independently before Multimodal Fusion (MF) training, the training framework does not allow a stage where all architecture components are trained end-to-end. This might hinder the model’s ability to create representations and encode all relevant information between both modalities, since the unimodal encoders are never updated together with the fusion architecture component. This may be difficult to address during a short rebuttal period, however discussion would be welcome regarding the training of individual components of the architecture: For example, why does the multimodal BrainHarmonix model underperform the individual performance of BrainHarmonix-F on the ADHD-200 dataset (in terms of F1-score), when in theory it has access to more information.\nParameter Count: Regarding the performance of BrainHarmonix versus individual modality components, additional discussion would be welcome on how much of the performance might stem from increased parameter count in the full BrainHarmonix model (with the Harmonizer model component) versus performance increase from having additional data modalities to perform downstream tasks.\n\nQUESTIONS:\nDo the authors have evidence that the latent space of brain representations learned by their model reflects structural constraints more than previous brain foundation models as a result of the incorporation of geometric harmonics?\nCan the authors clarify details of the TAPE encoding scheme, mainly (i) how the linear transformation matrices are learned and (ii) how much information loss there is (if any) when creating tokens across datasets with varying repetition time.\nCan the authors comment on end-to-end training versus modular training as described in their framework?\nHow much performance of the full multimodal model comes from the inclusion of multiple data modalities versus increased parameter count from having an additional architectural module (the harmonizer)?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}