BrainMoE: Cognition Joint Embedding via Mixture-of-Expert Towards Robust Brain Foundation Model
Authors: Ziquan Wei
Tingting Dan
Tianlong Chen
Guorong Wu
Venue: NeurIPS 2025 poster ()
Keywords: Mixture-of-Expert
Brain foundation model
Large model
Semi-supervised learning
Transformer decoder
Primary area: neuroscience_and_cognitive_science

Abstract: Given the large scale of public functional Magnetic Resonance Imaging (fMRI), e.g., UK Biobank (UKB) and Human Connectome Projects (HCP), brain foundation models are emerging. Although the amount of samples under rich environmental variables is unprecedented, existing brain foundation models learn from fMRI derived from a narrow range of cognitive states stimulated by similar environments, causing the limited robustness demonstrated in various applications and datasets acquired with different pipelines and limited sample size. By capitalizing on the variety of cognitive status as subjects performing explicit tasks, we present the mixture of brain experts, namely BrainMoE, pre-training on tasking fMRI with rich behavioral tasks in addition to resting fMRI for a robust brain foundation model. Brain experts are designed to produce embeddings for different behavioral tasks related to cognition. Afterward, these cognition embeddings are mixed by a cognition adapter via cross-attention so that BrainMoE can handle orthogonal embeddings and be robust on those boutique downstream datasets. We have pre-trained two existing self-regressive architectures and one new supervised architecture as brain experts on 68,251 fMRI scans among UKB and HCP, containing 12 different cognitive states. Then, BrainMoE is evaluated on a variety of applications, including sex, age prediction, human behavior recognition, disease early diagnosis of Autism, Parkinson's disease, Alzheimer's disease, and Schizophrenia, and fMRI-EEG multimodal applications, where promising results in eight datasets from three different pipelines indicate great potential to facilitate current neuroimaging applications in clinical routines.

BrainMoE: Cognition Joint Embedding via Mixture-of-Expert Towards Robust Brain Foundation Model
Ziquan Wei
Tingting Dan
Tianlong Chen
Guorong Wu*
Departments of Computer Science and Psychiatry University of North Carolina at Chapel Hill Chapel Hill, NC 27599
(ziquanw, tianlong) 8cs unc edu; (Tingting _an, grww) unc.ddu
Abstract
Given the large scale of public functional Magnetic Resonance Imaging (fMRI) e.g., UK Biobank (UKB) and Human Connectome Projects (HCP), brain foundation models are emerging. Although the amount of samples under rich environmental variables is unprecedented. existing brain foundation models learn from fMRI derived from narrow range of cogmitive states stimulated by similar environments, causing the limited robustness demonstrated in various applications and datasets acquired with different pipelines and limited sample size. By capitalizing the variety of cognitive status as subjects performing explicit tasks, we present the mixture of brain experts, namely BrainMoE, pre-training on tasking fMRI with rich behavioral tasks in addition to resting fMRI for robust brain foundation model. Brain experts designed produce embeddings for different behavioral tasks lated to cognition. Afterward, these cognition embeddings are mixed by cognition adapter gross-attention so that BrainMoE can handle orthogonal embeddings and be robust on those boutique downstream datasets. We have pre-trained two existing self-regressive architectures and one new supervised architecture as brain experts 68, 28,251 1 fMRI scans among UKB and HCP, containing 12 different cog- nitive states. Then, BrainMoE is evaluated on variety of applications, including sex, age prediction, human behavior recognition, disease early diagnosis of Autism, Parkinson's 's disease, Alzheimer's 's disease, and Schizophrenia, and fMRI-EEG mul- timodal applications, where promising results in eight datasets from three different pipelines indicate great potential facilitate current neuroimaging applications clinical routines.
Introduction
Like foundation models for other topics, brain foundation models aim learn feature representation fundamentally from large-scale data of neuroimaging. Functional Magnetic Resonance Imaging (fMRI) of the brain, it offers insight into the relationship between functional fluctuations and human behavior [[, is critical to discovering the enigma of human cogmition and promoting clinical applications. BoooodOOygen-LLeve
Dependent (BOLD) signal in fMRI measures neuronal activity. Such raw signals are preprocessed as timeseries of regional mean or the functional connectivity (FC) by coefficient correlation for analysis with high Signal-to-Noise Ratio (SNR) (6). While the exploration of brain foundation model has expanded to various masking strategies for either (latent)
BOLD FC reconstruction [23 34] previous works formulating this problem as transferring
Cooressponiing author.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

(a) Pretrain on resting state
(b) Pretrain on mulliple
(c) Mffttre-of-EEpert
cognitive states
AX-Z)
Experts cognilion embedding
z
Z across cognitions
((::))
space
space
Latent space
Information bottleneck on single cognitive state
Figure 1: Motivation of BrainMoE through the lens of the information bottleneck theory. (a) Feature representation learning makes an information bottleneck between data and label space, where denotes data, Z. denotes latent feature representation, Y is the label of data, and ((::-) is the mutual information. (b) model pre-trained on multiple cognitive states may compromise the underlying heterogeneity between different states, where Z cannot be optimal for all states. (c) The mixture of brain experts dedicated to diverse cognitive states leads to a joint cognition embedding so that the downstream applications can be advanced by stratified pre-training on rich behavioral tasks.
self-regressive methodologies from natural language and image to neuroimaging ignored the inter- correlation between non- oon-imginn phenotypes [1. Furthermore, they are restricted by narrow range of one or two cognitive states, e.g., the resting state, causing samples with behaviors other than resting to be overlooked. Due to the lack of explicit designs utilize the complete brain fMRI dataset with respect to neuroscience knowledge, a mixture of brain experts is proposed towards a robust brain foundation model in this work.
UK Biobank (UKB) (20) and two Human Connectome Projects (HCP) (29 5 that contain healthy subjects 22 to 100 years of age are mainly used as pre-training datasets since they have large scale. Previously, most subjects in resting state among UKB and HCP were included in BrainMass [34 and BrainJEPA | While BrainLM (23 involved an additional tasking state in UKB, its performance has shown worse than others. Even though BrainMass has collected the most available published resting fMRI data on the OpenNeuro platform [22]. ten available cognitive states in HCP datasets were ignored. It is intuitive to train single model with all available data. However, as shown in Fig. (a) and (b), simply pre-training with all cognitive states results single model being suboptimal to samples with different cognitive states, e.g., the red information bottleneck established by mutual information suboptimal in the latent space where cognition related behaviors are variable. In fact, single model is observed to compromise the underlying heterogeneity between cognitive states derived from diverse neural circuits stimulated by different behaviors 255. This issue necessitates mixing experts specialized in different cognitive states, as shown in Fig. (c), where each expert produces the cognition embedding, that is, a feature representation stratified by cognitive states.
On the other hand, tremendous efforts have been made to benchmark generally purposed models [8] on brain fMRI data. common observation on the results is that performance is diverse using BOLD or FC as the model input for different datasets, leading to related brain fMRI analysis works being categorized by types of input: (1) BOLD foundation models [23 [1] and (2) FC foundation models [15] 34 Nevertheless, this reduces the adaptability of previous brain foundation models for datasets that better with a type of input differentiated from the pre-training stage. Mixture-of-expeee
24 10 and brain dedicaedd architectures 19
(MoE) cooperating with router and adapter [35] 37 has demonstrated great potential for multimodal, referring to BOLD and FC in our data, and multitask, referring multiple cognitive states. Therefore, novel cognition adapter is proposed to facilitate BrainMoE as a robust brain foundation model learning from various cognitive states. Although adapters in MoE for language and vision fields are using small architecture like a multilayer perceptron (MLP) [8], a high scalability of the cognition adapter can ensure the transformation from cognition embeddings to objectives. Given that MLP is not scalable (see Appendix), a Transformer decoder is utilized for adapting BrainMoE to downstream applications.
To this end, this work has three contributions: (1) We propose BrainMoF, an MoE framework for brain fMRI data that towards high robustness for downstream tasks with different pipelines and limited sample size. (2) A cognition adapter is designed to adapt embeddings from experts pre-trained

---

ABIDE (,05))
PPMI (x=209)
HCPA (wx4,863)
HCPYA (v=3.99)
69.5
62
97
69
score
68.5
F1 score
61
F1 soore
88
68
60
67.5
67
59
F1
89
66.5
58
87
78
30k
Pre-train data
Pre-train data #
Pre-train data #
Pre-train data #
BrainMass BrainMoE
BrainMass
BrainMoE
BrainMass
BrainMoE
BrainMass
BrainMoE
76
52
55
76
66
F1 soooe
47
scote
50
71
42
45
61
56
96
Pre-train data # Ft
51
32
35
46
30k
68k
30k
6Bk
30k
30k
68k
Pre-train data
Pre-train data #t
Pre-train data # BrainMoE
BrainJEPA BrsinMoE
BrainJEPA BrainMoE
BrainJEPA
BrainJEPA BrainMoE
Figure 2: Increasing the scale of pre-training data of brain foundation model leads marginal or negative performance boost due to 11 overlooked cognitive states. Four columns are four down- stream applications, where ABIDE and PPMI are disease recognition, and HCPA and HCPYA are human behavior recognition. Two rows are two expert architectures, where BrainMass uses FC and BrainJEPA uses BOLD.
with various cognitive states, regardless of the input type, for finetuning. (3) Two existing brain foundation models pre-trained in self-supervised manners and one cognition classifier pre-trained in a supervised manner are both evaluated as experts in BrainMoE, where experts pre-train on 68,251 fMRI scans among UKB and HCPs and fine-tune on various applications, including sex prediction, human behavior recognition, and disease early diagnosis of Autism, Parkinson's disease, Alzheimer's disease, and Schizophrenia among seven datasets.
Preliminaries
Brain foundation models To the best of our knowledge, BrainLM [23 represents the first brain foundation model. It applied Masked Autoencoding (MAE) to BOLD signals reconstruction. How- ever, densely filling the entire IMRI time series can impair the model's capacity to differentiate between noise and meaningful signals. Prior work [3] has demonstrated that masked pretraining in generative frameworks such MAE often yields suboptimal results in off- fff-the-shelf evaluations, such as linear probing. Similarly, BrainJEPA [I]] introduces an alternative architecture employing distinct JEPA-basee masking strategy, addressing BrainLM's limitations by drawing insights from I-JEPA [3] Although BrainJEPA reports superior performance relative to linear probing. it does not explicitly incorporate pre-training with tasking fMRI. BrainMass 34 used larger pre-training dataset (see Appendix) and a matching objective between pseudo FC matrices as a novel framework. Whilst, used sololy the resting fMRI and overlooked more than 38k tasking fMRI in the dataset.
Resting- and tasking-state fMRI Neuroimaging data contains brain activity reflecting the interac- tion between functional fluctuations and human behavior. Studies controlled subjects in a resting explicit tasking state during the data acquisition offer distinct, complementary perspectives on brain function [36]_ Large-scale studies (29 5 20 are observed to collect at least one tasking state in addition to the resting state, while brain foundation models mainly pre-train on the portion under the resting state, overlooking half or more data in the datasets. The intuitive method of mixing all data together may compromise the underlying heterogeneity between different states. Fig. shows that the improvement gained from pre- traiiiain with 38k more data containing 11 overlooked cognitive states marginal on four downstream classifications: ABIDE 2-class classification for Autism diagnosis, PPMI is 4-class for staged Parkinson's 's disease, HCPA is 4-class, and HCPYA is 7-class for human behavior recognition.
contrast, BrainMoE. can bring an impressive performance enhancement by stratifying and adapting cognitive states. Note that ABIDE and PPMI are preprocessed by the pipeline of 33), which different from our pipeline (see Appendix) for HCPs and UKB datasets. Furthermore, BrainMass and BrainJEPA reconstruct FC and BOLD latent features, respectively.

---

Reconstruction
(2) Classification
Experta1
Cognition
enbeds
restng??
Expert#1 $
Cognition enbeds
FC OR BOLD
Expert#22
Cognition
VISMOTOR?
Jask-
FC OR BOLD
Expert#2
Cognition Adapten
Routen
Pre-training
ExpertiN
Cognition emibdds
Jask-
FACENAME?
Fine-tuning
ExperteN
Cognition enbeds
Autripie
Cogni -tion embed
Cogni -tion embed
Multiple Loyors
Multi- Head Cross- Attention
Copy
Q
Cogn1
Cogn1 enbed
Any nodel architecture
query embed
query embed
embed
15.
Supervision
Legends
FC
Suparvision
OR
Classifiatioo
Learnable
(c) Brain Export
Reconsinaction
Veriables
Frozen
(d) Cognition Adapter
BOLD
Figure 3: Framework of BrainMoE. (a) Pre-training has two options: (1) Previous models train with
reconstruction objective, where samples used for BrainMoE pre-training are stratified by cognitive states. (2) A new architecture of cognition classification can be set as the objective via cross-entropy between embeddings and cognitive states. (b) Fine-tuning a router for the expert selection and
cogmition adapter for the combination of frozen experts.
(c) Brain expert is adaptable to any model architecture, learning feature representation from either FC or BOLD signal, where the vector of latent feature before the final predictive layer is used as a cognition embedding.
(d) Cognition adapter is Transformer decoder, where MHSA stands for multi-head self-atte attention, and FFN is feedforward network.
These primary results demonstrate empirical and clear evidence to support the motivation to stratify data according cognitive states for multiple experts and learn joint cognition embeddings.
Methods
BrainMoB: is designed to work with arbitrary brain foundation models experts and cooperate with a router and an adapter for fine-tuning on the downstream. Assume the input, BOLD or FC, is denoted by M xC with M regions of the brain atlas and Cin channels of input vector. Target of brain experts is to produce cognition embeddings, Z. For the router and adapter, it is to predict Y for downstream applications given pre-trained experts.
3.1 Framework
The framework of BrainMoE is separated into two stages as shown in Fig. (a) pre-training and (b) fine-tuning, where N experts, denoted by f(-) XCC. gCass with Clad the hidden channel number, leam from large-scale datasets containing subjects explicitly tasking N cognition-related behaviors to produce variety of cognition embeddings, and fine-tune a router for expert weights Pe R.N for selecting top-k (k N1) experts, and a cognition adapter for predicting downstream tasks based on cognition embeddings. Following the observed performance that relies on input type, preprocessing pipeline, and model architecture, BrainMoE has framework suitable experts with no requirement for data type and architecture.
3.2 Expert pre-training
In Fig. ()), two objectives can be used to pre-train the brain expert, the reconstruction and the classification. (1) The pre-training of existing brain foundation models is reconstructing latent feature of input, FC or BOLD, from its masked version via bottleneck or transformer encoder architecture. We utilized BrainMass BrainJEPA candidates of brain experts. The latent feature is produced by

---

existing architectures as the cognition embedding, denoted by Z E R Each expert is pre- pre-trained with the data that has the same cognitive state.
== fX) arg min |Z (x)]l?
(I)
where g is the target network. As shown Fig. 3 (c), the brain expert can any architecture that produces latent feature representation, which frozen and copied for the downstream. The pre- pre-training objective is not restrict to the reconstruction or a new classification for cognitive states.
In Fig. 3 (a) (2), we propose new pre-training objective, cognitive state classification, to explicitly learn from the cross-entropy between the latent feature and the cognitive state, CF..oss(p(Z.) Yoog, where p : RCasa IR 1 is linear layer, and Yogg is the binary label of cognitive state. The architecture for this expert is the same as the cognition adapter introduced in the next section.
3.3 Cognition adapter fine-tuning
The architecture of the cognition adapter is designed Transformer decoder shown in Fig. (d). The purpose of this adapter is to adapt multiple experts from a stratified feature representation based on cognitive states downstream application, classification task in this work.
Assume that the token vectors shown in dashed rectangle in Fig. (d) is denoted by Z E IR+ Plx
where the class number in the downstream application. Note that Z. Z c P representing
the top-k cogmition embeddings from experts and Zk:(k++)) P) denotes randomly initialized task query embeddings. It is also cognition classifier without Z.k. Then, demonstrated in the architecture, layer of the adapter starts at multi-head self-attention (MHSA), = Sof coftmax((e foftmax((K" K /OT//VO /Cria) with following definitions
(QK"))))))
(2) where āi; 7n E R Caia are learnable parameters, and h is the head index. Last, multi- head cross-attention brings the information from the raw input to the task embeddings. Suppose Ie R MxM M is FC matrix.
Q == Zā,, == 23,, Z7
Chia
between Z and I with alternative definitions
Q == Iå,, K == ZB,, V := 16
where ai. Y E R MxChia Br IRCnaaCChia are learnable parameters. FFN denotes a feedforward network constructed by MLP. Note that the bias in linear layers is omitted in this section for clarity. Finally, after multiple layers of the cognition adapter. linear layer, I PxChnu R Px1 takes only the task query and produces the logistic prediction to accomplish fine-tuning on a downstream application.
Experiments
We evaluate the proposed BrainMoE on 3 pre-training datasets, including UK Biobank (UKB), HCP Aging (HCPA), and HCP Young Adult (HCPYA), and 7 downstream datasets, including ADNI, ABIDE, PPMI. Taowu, SZ, HCPA, and HCPYA. UKB and two HCPs contain 68,25 scans of brain fMRI from 21,797 subjects under 12 different cognitive states on resting or tasking. Five disease-related datasets contain more than 1,500 subjects under the same resting state but various health status.
To comprehensively evaluate and showcase the performance, we conduct experiments on both randomly initialized and pre-trained models across tasks involving disease, sex, and brain state recognition. Specifically, our study aims to address two key research questions: (RQ1) To what extent does BrainMoF. improve the prediction performance from the baseline using different expert architectures? (RQ2) Which pre-traine training objectives are the most robust across various downstream applications? Additionally, we provide ablation studies further support our findings.
Datasets We preprocess fMRI and partition brain regions using the AAL atlas (8 for UKB, HCPA, HCPYA, and ADNI. Details can be found in the Appendix. Other datasets preprocessed by [3] SZ is an in in-house data preprocessed by a third party.
UK Biobank (HCPA) dataset 20 is large-scale dataset with MRI data. There are fMRI (n=51,780)
involved this work. It consists of one resting state and one tasking state that engages cognitive and sensory-motor [3].
5

---

BrainMoE
BrainMass
Resting
Hariri
ADNI (AD)
Working memory
ABIDE (ASD)
Motor
Gambling
PPMI (PD)
Facename
Taowu (PD)
Language
HCPA (3-task)
Social
HCPYA (7-task)
Relational
Emotion
SZ (SZ)
Caril
Vismotor
60
80
100
100
10"
10"
Phenotype
Sex
PT size
Figure 4: The performance of BrainMoE and BrainMass pre-trained with all samples, and n=12 pre-trained individual experts on phenotypic and sex classifications among 7 datasets, where scores lower than 50% are hidden for clarity, and the pre-training (PT) size ranges from 68,251 to 637.
The Lifespan Human Connectome Project Aging (HCPA) dataset [5] is instrumental in task recognition research, offering a comprehensive view of the aging process. It includes data from 717 subjects, encompassing IMRI records (n=4,863) with four human behaviors associated with memory. sensory-motor, and the resting state.
The Human Connectome Project Young Adult (HCPYA) dataset (29 has tackled key aspects of the neural pathways that underlie brain function and behavior via high-quality neuroimaging data in over 1, 100 healthy young adults. It includes data from seven human behaviors associated with various cognitive tasks, e.g., language and working memory.
Alzheimer's 's Disease Neuroimaging Initiative (ADNI) dataset 32 serves as an invaluable resource, featuring a collection of pre-processed IMRI (n=138) and including clinical diagnostic labels. It encompasses spectrum cognitive states: Cognitive Normal (CN), Subjective Memory Complaints (SMC), Etrly-Stage Mild Cognitive Impairment (EMCI), Late-Stage Mild Cognitive Impairment (LMCT), and Alzheimer's Disease (AD). Considering the class unbalance issue, we simplified these categories into two broad groups based on disease severity: combined CN, SMC, and EMCI into 'CN' group, while LMCI and AD were grouped as the 'AD' group.
Parkinson's 's Progression Markers Initiative (PPMI) dataset 33 presents substantial collection of data from 209 subjects. encompasses states of mental health: normal control, scans without evidence of dopaminergic deficit (SWEDD), prodromal, and Parkinson's disease (PD).
Taowu one of the earliest image datasets released for Parkinson's 's and contains 40 subjects.
presents data from 1,025 young adults. The initiative aggregated fMRI data collected from laboratories around the world support the research on Autism Spectrum Disorder (ASD).
Autism Brain Imaging Data Exchange (ABIDE) dataset 33
Schizophrenia (SZ) is the in-house data that contains 189 subjects. There are 30 converted and 159 nonconverted.
Implementation Following previous works, our experiments are conducted with subject-level cross-validation (CV). The average score and the standard deviation are both listed. To make our results comparable with previous papers, HCPA, HCPYA, and ADNI use 5-fold CV as same as [1, while others use 10-fold same as [33. Since HCPs used for both pre-training and fine-tuning, the training data the two stages is always from the corresponding CV fold's training set to prevent data leakage. Hyperparameters, e.g., learning rate and hidden channels, can be found in the Appendix.
State-of-ther ahe (SOTA) brain foundation models, BrainMass 34 and BrainJEPA [I], are selected as expert architectures along with the new classifier architecture proposed in this work. Note that the original BrainMass fine-tuning utilizes the support vector machine (SVM) that has lower scale of learnable parameters than others. Therefore, an enhancement of Brain evaluated this work by replacing SVM with 2-layer MLP.

---

Table 1: MoE improvement on phenotypic classification Fl score compared to the baseline, where 30k is pre-trained on resting-state data (m=29,951), and 68k is pre-trained on all data (n=68,251), PT stands for pre-training. Colored text indicates the performance increase/decrease from using 68k.
BrainMass
BrainJEPA
Predictor SVM
SVM
MLP
MLP 68k
BrainMoE
VIT
VIT
BrainMoE 68k
PT
30k
68k
30k
68k
30k
68k
ADNI
75.32
7..32
76.8617.26 7
80.70
=1.23
74.1648.55
74.1646.53
77.11 77.11 26.64
LAD
0.00
3.84
4.37 f
0.00
2.95 1
ABIDE 61.91
64.12-2.31
66.81 44.18
67.81 #3.91
70.26=3.40
36.25
54.55 89 39.82+3.91
LASD
1.81
1.00
3.45 1
3.77
18.30
PPMI
44.87
56.52_14.86
58.90
59.77
62.97 =
38.69
38.69413.91
60,49
LPD (staged)
1.65
0.87
4.07
0.00
21.80
Taowu 58.33134.75
65.67
70.29 177.97
68.00
88.57 12.51
36.08 126.38
36.94
79.86 14.440
LPD (hinary)
7.34
2.29
18.28 1
0.86
43.78
SZ 96.95
76.95=a.01
79.85
77.63+8.56
83.10=11.33
76.98+9.00
78.97
82.86+9.11 19
L,Shiiophremia
0.00
2.22
3.25 f
1.99
5.88 f
ICPA 85.16
89.73+0.58
87.91
90.63
96.67 59.54
53.12
81.74
4.3ttskkrest
4.57
2.72
8.76
6.42
22.20
HCPYA 77.51
1.77
79.40
81.27 11.27
93.19-u.72
50.68 50.68 425.20
74.5943.79
47-task
3.36
1.87
13.79 1
5.42
23.91 1
4.1 RQ1: MoE vs baselines
The average Fl scores of BrainMoE and BrainMass pretrained with all samples, and n=12 individual experts per cognitive state phenotypic and sex classifications among datasets are shown in Fig. 4 Previous studies have demonstrated good PT data scalability with resting-state fMRI data. However, according to Fig. + there are consistently existing task-specific experts (e.g.. language for AD, working memory for PD) outperforming Resting experts, confirming that task-state fMRI contains valuable information for brain modeling. Conclusively, BrainMoE holds the best performance compared to all experts across 7 datasets. This supports that the utilization of cognitive embeddings from BrainMoE leads more robustness of brain modeling than naively training single task of fMRL.
In Table D we summarize the impact of BrainMoE on downstream phenotypic classification, reporting improvements the Fl score relative to non-MoE baselines for disease and human behavior recognition. Across all tasks (ADNI, ABIDE, PPMI, Taowu, SZ, HCPA, HCPYA), intuitively expanding pre-training data from 30k to 68k yields modest gains, even negative gains, for both BrainMass with SVM and MLP, c.g. ABIDE BrainMass SVM: +1.81 Fl and SZ BrainMass MLP: -2.2 F1, and BrainJEPA with Vision Transformer (ViT), c.g., HCPA BrainJEPA: -6.42 F1. In contrast, introducing BrainMoE with the proposed cognition adapter on top of the 68k pre-trained backbone amplifies these gains substantially: phenotypic F1 score uplifts range from +3.25 F1 (SZ Schizophrenia) +43.78 F1 (Taowu PD), and it consistently brings positive effect. Even BrainJEPA baselines that do not gain F1 improvement from more data on HCPA and HCPYA benefit from BrainMoE, albeit lesser extent (e.g., HCPYA: +3.04 FI). Notably, the largest relative benefit appears on smaller cohorts, e.g., Taowu (n=40), where BrainMoE achieves + +18.28 and +43.76 F1 over 68k BrainMass and BrainJEPA baselines.
Table 2 reports analogous results for sex classification. Worse than phenotypic classification, in- creasing pre-training size delivers small, commonly improvements for BrainMass and BrainJEPA using SVM, MLP, and VIT, where 11 out of 18 experiments have dropped Fl scores in red text. In contrast, BrainMoE recovers and exceeds prior performance. It consistently demonstrates Fl score gains, except for BrainJEPA on ABIDE. is worth noting that the most dramatic uplift appears on the smallest dataset (Taowu, 4==0), where BrainMoE increases F1 by +43.76 76 BrainJEPA.
Overall, the above results demonstrate that (1) a large scale pre-training without stratifying cognitive states improves downstream performance modestly (sometimes negatively), and (2) the proposed BrainMoB framework produces substantial gains, especially on downstream applications with limited sample size.
7

---

Table 2: MoE improvement on sex classification Fl score compared to the baseline, where the sample size of the downstream dataset is indicated. PT stands for pre-training.
BrainMass
BrainJEPA
Predictor
SVM
SVM
MLP
MLP
BrainMoE
ViT
ViT 68k
BrainMoE
PT #
30k
68k
30k
68k
68k
30k
68k
ADNI
48.60 6655
54.30
12.48
64.82 14.30
59.30
13.05
69.22 15.22...22
37.70 18.73
36.42 ja.22
62.98 62.98 77.76
L,==138
5.70
5.52
4.40 f
1.28
25.28
ABIDE
73.84_3.49
73.84
75.12
85.21
78,08
78.08+5.84
78.08
Ln==0225
0.00
0.00
10.09
0.00
0.00
PPMI
52.03
14.03_1..6666.583
12.228
12.28
64.73
14.
79.8118.48
46.23
46.2319.00
67.57
Ln=209
4.55
1.41
16.49
0.00
21.34
Taowu
46.24_23.90
46.24
23.96
62.86
88.28.220
55.38
_220000000
74.00
46.24
51.667
51.67 121.1221
90.00 12.000 12.96
Ln=40
0.00
7.48
11.14
5.43
43.76
HCPA
66.20
68.25
66.93
68.58
76.76.2..9
40.32
40.32
44.24
L,x=48633
2.05
1.65
9.83 f
0.00
3.92
HCPYA
63.33_3.01
65.47 #3.51
64.57 22.03
66.98 3.330
74.3644.43
40.20
40.20
43.24
Ln==293
2.14
2.41
9.79
0.00
3.04
Router Probability (%)
Router Probability (%)
Pearson coefficient correlation between experts
Emot
Errot
Carl
Emol-W/
Car-Fore
Face
Harti
-0.10
Emor-Moto
Face
Gamb-Rela
Gamb
30
-0.02 0.22
Face-Rala
Motor
Lang- aan--Face acce
Lang
25
Gembling
-0.11 -0.27 -0.20
Gamb-Reis
Long Role
Lang
Lang-
Moto
20
Faceneme
Rela
0.22 -0.17
0.15
0.44
-0.12
Moto
Rola
Language
Moto-Hari
Rala
Rest-Gemb
15
0.28
.0.05
0.51
0.06
0.01
-0.14
Rest
Rest-Molo
Rest Emol
Reld
-0.09
-0.02
-0.10
-0.24
0.16 -0.19 -0.04
Rast-Mtto
Soc--Rala
Rest-Rals
Reat-Hri
Rest-WI
Soci
10
Vism- Rolo
WM/-Face
Vism-W/1 W/M
San
-5
10
Relational
Emotion
Carit
0.34
-0.23
0.34
-0.11
Whl-Har
0047 0.02 40.28 0.28 0.12 -0.22 0.04 -0.30 0.20 -0.14
WM--Moto
Wi--Molo
WM-RRea
WIM Sop
0.47 0.00 -0.31
12 -0.04 0.13 0.00 0.38 0.21 -0.23 0.02
ADNI
ABICE PPM
Real
Harii
Moter
Gambling
Facaname
Language
Siocial
Reletional
Emoteen
Cant
(a)
(b)
(c)
Figure The distribution of dominating expert combinations (a) late fusion MoE and (b)
BrainMoB shows the router preference, where the first 4 letters of cognitive state are used as abbreviation. (c) The correlation between cognition embeddings experts shows expert diversity.
4.2 RQ2: Pre-training objectives
As described in See the BrainMoE framework has no requirement for data type and architecture. This property results various pre-training objectives for the brain expert: FC reconstruction (FC recon.), BOLD reconstruction (BOLD recon.), and cognitive state classification (Cog. Classif.). To demonstrate which objectives are the most robust, we evaluated three objectives and an all-in-one BrainMoE on downstream datasets in Table and
Briefly, FC reconstruction and all-in-one BrainMoE show the best robustness. They always rank in the first two places for phenotypic (Table 3 and sex classification (Table across all downstream datasets, except for the smallest dataset Taowu (n=40). Unlike BrainMoE has the cognition adapter to implicitly utilize expert embeddings, the late fusion explicitly combines expert predictions, therefore cannot handle the unbalance issue. We can observe in Table that FC reconstruction shows the most first rank, out 7 datasets, and all-in-one has the most first place, out of in Table Although top-k selected experts in all-in-one BrainMoE contain experts in FC reconstruction, the self-attention in the cognition adapter mixes information between all types of pre-training objectives, yielding dropped and boosted performance for phenotypic and sex classification, respectively.

---

Table 3: MoE performance on phenotypic classification Fl score using three types of expert pre- trained with three objectives, along with an all-in-one MoE mixing all types of experts, where LF is Late Fusion, Ex. # denotes expert number. Bold is the first rank and underline is the second.
Ex. # ADNI
ABIDE
PPMI
Taowu
sz
HCPA
HCPYA
Baseline
BrainMass
80.70
67.81
14.77 68.00 77.6348.56
90.634a.74
81.27
BrainJEPA
16.16.
39.82+3.91
38.69
36.94
78.97
53. 12.1111
56. 56.10429.u
LF-MoE
73.33
69.89
61511
91.24 49.50
94.9642.64
43.38
BrainMoE
FC recon.
12
81.23
70.26
62.97
13.94
83. 83.10 88.57
96.67.10.77
93.1940.
BOLD recon.
12
54.55 77.11
60.49
79.86414.46
81.74 82,8649.19
74.59+3.79
Cog. classif.
12
10..2
68.65
33.81
90.48 41.23
83.36
96.28+a.70
95,81 80.44.088
All-in-one
36
19.73+10.60 49.03 60.76+14.86 85.93
83.91
96.6640.94
96.81+0.41
Table 4: MoE performance on sex classification F1 score using three types of expert pre-trained with three objectives, along with an all-in-one MoE mixing all types of experts. Bold is the first rank and underline the second.
Ex.
ADNI
ABIDE
PPMI
Taowu
HCPA
HCPYA
Baseline
BrainMass
59.30=13.05
75. 6..32
64.73+44.12
55.38
68.58+0.99
66.98
BrainJEPA
36.42 1..22
78.08 15.84
46.23 19.00
51.67121.12
40.3214.07
40.2014.22
BrainMoE
FC recon.
12
69 69.22 11/22+6.20 5.20
85.21 3.777
79.81 +8.46
74.00
76.76+0.93
74.36+44.43
BOLD recon.
12
62.98 77.76
78.08 16.16
67.57 77.24
90,00 12.96
44.24+
43.24
Cog. classif.
7..61
82.82+5.11
79.07 +7.28
72.22+25.17
75.65
75.18+1.16
All-in-one
36
70.72.7.58
82.85
82.80
75,29 130.56
78.3412.18
77.67
4.3
Rounter and expert analysis
The preference of routers in late fusion MoE and BrainMoE is shown in Fig. (a) and (b), respectively, where the percentage indicates how many samples have combination experts with dominating router logits (> N Clearly, BrainMoE has diverse and similar dominating combinations for heterogeneous and homogeneous applications, respectively. The combinations are mainly dual, and datasets with the same task share similar pattern (i.e., cognitive state for HCPA and HCPYA, and PD for PPMI and Taowu). In contrast, late fusion consistently has single dominating expert due to data scale diversity, e.g., Rest (n=29,971), which implies that the routers trained by BrainMoF learned more neuroscientific knowledge than the late fusion. Furthermore, the investigation on expert embeddings is shown in Fig. [c). The absolute value of correlation is mostly less than 0.5, indicating experts are not conflict or redundant to each other.
4.4 More applications
The age regression across 6 datasets has been evaluated for the best baseline, 68k version of BrainMass, and BrainMoF, as
Table 5: Age regression performance compared to the baseline, where the sample size of the downstream dataset indicated, and unit is year.
MSE ADNI
ABIDE
PPMI
Taowu
HCPA
HCPYA
BrainMass 36.28 19.83 36.77 33.09 38. 10 27.66 5,46 2.
BrainMoE 36.27 10.23 12.67 29.89 30.72 12.25 10.56 1.566...66. 3.45 1.
listed in Table We can observe the performance of BrainMoE is the best, where the improvement is especially significant for ABIDE 1n=5, 6-58 yrs) with MSE 36.77 4.86. This empirical evidence further supports the robustness of BrainMoE.
Table 6: BrainMoF. applies on a multimodal dataset, NATVIEW [27]. NATVIEW
Multimodal applications of cognitive state, sex classifications, and age regression in an fMRI- EBG dataset [27] are listed in T There are 388 fMRI-EEF MMRI-EEG pairs from 22 healthy subjects F.M=1:1).
8-task (F1) (F1) Age (MSE)
BrainMass (fMRI) 67.66 63.67 55.1 8.05 =5.5
under 8 cognitive states
CBraMod (EEG) 68.7 + 1.46 25.39 8.26+6.97
+2.33
BrainMoE is evaluated here with n=1 pre-
BrainMoE (13 Ex.) 68.73 3. 65.47 : +5.38 7.99 55.53

---

Phenotypic classification (12-expert)
Phenolypic classification (36-experl)
Sex classification (12-expert)
Sex classsfication (36-expert)
100
90
90
80
80
F1 score
F1 11o0rre
75-
70-
60
66
Below 50%
Below 60%
50
50%
100%
50%
100%
50%
100%
50%
100%
k=
ABIDE PPMI
HCPA
HCPYA
BrainMass
BrainJEPA
BrainMoE
Figure 6: Impact of altering k on downstream classification performance. Mean F1 scores with standard deviation are reported for four downstream datasets under three expert-selection regimes (top-1, top 50%, and all experts). Colors represent different datasets.
trained CBraMod (30 expert plus n=12 cognition classifier experts (Shaefer400 version). Results from a 5-fold CV show the best performance by multimodal BrainMoE.
4.5 Ablations
Top-k We alter l in BrainMoE to strictly limit how many top experts can be selected for downstream applications. We evaluated with relatively larger sample sizes in Fig. 6 where 12-expert BrainMoE uses BrainMass as the expert architecture. From the left two panels, is obvious that increasing ie has a slight difference for all phenotypic tasks, except for HCPA. In the right two panels, we can observe that sex classification benefits less from a larger k. Overall, expert scaling BrainMoE yields clear benefits for complex, data-scarce phenotypic tasks, with most gains achieved by employing top half of the experts. Beyond this, adding experts delivers diminishing returns. contrast, for the simpler binary sex classification task, especially with large expert pool, additional experts do not meaningfully improve performance and may introduce redundancy or overfitting. Thus, tailoring the number of active experts to task complexity and dataset size is key for efficient MoE deployment.
1, A 50%, and k 100% with 12 and 36 experts on four datasets
Conclusion
In conclusion, we propose a new framework of the brain foundation model, BrainMoE, to pre-train with overlooked tasking-state fMRI for robust downstream applications. We observe that existing brain foundation models learn from fMRI derived from a narrow range of cognitive states, while there are 11 available cognitive states as subjects performing explicit tasks in large scale datasets. Furthermore, we showcase that (i) the straightforward utilization of data with rich human behavioral variables by pre-training with all data and (ii) the late fusion MoE both improve performance marginally. Aiming at these challenges, BrainMoE pre-trains each expert on a portion of the datasets with the same cognitive state among 12 different states for a robust brain foundation model. We design a scalable cognition adapter to mix brain experts for downstream fine- fine-tuning so that BrainMoE can handle orthogonal cognition embeddings and be robust on the boutique downstream datasets. With sufficient 68,251 pre- pre-training fMRI scans among UKB and HCP with 12 different cognitive states, BrainMoE has shown impressive performance boosting on a variety applications, including sex, age prediction, human behavior recognition, multimodal applications, and early diagnosis of various brain diseases. The promising results demonstrated on eight datasets from three different pipelines indicate great potential to facilitate current neuroimaging applications in clinical routines.
10

---

Acknowledgement
This work was supported by the National Institutes of Health (AG091653, AG068399, AG084375) and the Foundation of Hope. Tianlong Chen was partially funded by the National Institutes of Health (NIH) under award IR01EB037 1011011111 The views and conclusions contained in this document are those of the authors and should not be interpreted representing the official policies, either expressed or implied, of the NIH.
References
[1] Analucia A Alegria, Joaquim Radua, and Katya Rubia. Meta-analysis of fmni studies of disruptive behavior disorders. American Journal of Psychiatry, 173(1)):111--1130 2016.
[2] Emily J Allen, Ghislain St-Yves, Yihan Wu, Jesse Breedlove, Jacob Prince, Logan Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, et al. A massive 7t finii dataset bridge cognitive neuroscience and artificial intelligence. Nature neuroscience, 26 2022.
[3] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Sulf-supe learning from images with joint- embedding predictive architecture. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15619- 15629, 2023.
[4] Hasan A Bedel, Irmak Sivgin, Onat Dalmaz, Salman UH Dar, and Tolga Çukur. Bolt: Fused window transformers for fmri time series analysis. Medical Image Analysis, 88:102841, 2023.
[5] Susan Y Bookheimer, David H Salat, Melissa Terpstra, Beau M Ances, Deanna M Barch, Randy Buckner, Gregory Burgess, Sandra W Curtiss, Mirella Diaz-Santos, Jennifer Stine Elam, al. The lifespan human connectome project in aging: overview. Neuroimage, 185:335-348, 2019.
[6] César Caballero- Gaadess and Richard C Reynolds. Methods for cleaning the bold fmri signal. Neuroimage, 154: 149, 2017.
[7] Li Chen, Patrick Bedard, Mark Hallett, and Silvina G Horovitz. Dynamics of top-down control and motor networks in parkinson's disease. Movement Disorders, 36(4):916-926) 2021.
[8] Hejie Cui, Wei Dai, Yanqiao Zhu, Xuan Kan, Antonio Aodong Chen Gu, Joshua Lukemire, Liang Zhan, Lifang He, Ying Guo, and Carl Yang. Braingb: benchmark for brain network analysis with graph neural networks. IEEE transactions on medical imaging. 42(2):493-500 2022.
[9] Tingting Dan, Jiaqi Ding, Ziquan Wei, Shahar Kovalsky, Minjeong Kim, Won Hwa Kim, and Guorong Wu. Re-think and re-design graph neural networks in spaces of continuous graph diffusion functionals. Advances in Neural Information Processing Systems, 36:59375-59387, 2023.
[10] Jiaqi Ding, Tingting Dan, Ziquan Wei, Hyuna Cho, Paul Laurienti, Won Hwa Kim, and Guorong Wu. Machine learning on dynamic functional connectivity: Promise, pitfalls, and interpretations.
arXiv preprint arXiv:2409.1
2024.
[11] Zijian Dong, Ruilin Li, Yilei Wu, Thuan Tinh Nguyen, Joanna Su Xian Chong, Fang Ji, Nathanael Ren Jie Tong. Christopher Li Hsian Chen, and Juan Helen Zhou. Brain-jepa: Brain dynamics foundation model with gradient positioning and spatiotemporal masking. arXiv preprint arXiv:2409. 2024.
[12] Michael D Fox, Abraham Snyder, Justin L Vincent, Maurizio Corbetta, David C Van Essen, and Marcus E Raichle. The human brain is intrinsically organized into dynamic, anticorrelated functional networks. Proceedings of the National Academy of Sciences, 102(27)9973-9677 2005.
11

---

[13] Ahmad R Hariri, Alessandro Tessitore, Venkata S Mattay, Francesco Fera, and Daniel R Weinberger. The amygdala response emotional stimuli: a comparison of faces and scenes. Neuroinage, 17(1):317-323, 2002.
[14] Tong He, Lijun An, Pansheng Chen, Jianzhong Chen, Jiashi Feng, Danilo Bzdok, Avram J Holmes, Simon B Eickhoff, and BT Thomas Yeo. Meta-matching as simple framework to translate phenotypic predictive models from big to small data. Nature neuroscience, 25(6):795- 804, 2022.
[15] Zhibin He, Wuyang Li, Yifan Liu, Xinyu Liu, Junwei Han, Tuo Zhang, and Yixuan Yuan. Fm-app: Foundation model for any phenotype prediction via fmri to smri knowledge transfer. IEEE Transactions on Medical Imaging, 2024.
[16] Mark Jenkinson, Christian F Beckmann. Timothy EJ Behrens, Mark W Woolrich, and Stephen M Smith. Fsl. Neuroimage, 62(2):782-790, 2012.
[17] Xuan Kan, Wei Dai, Hejie Cui, Zilong Zhang, Ying Guo, and Carl Yang. Brain network transformer. Advances in Neural Information Processing Systems, 35:25586-25599, 2022.
[18] Raymond Li, Gabriel Murray, and Giuseppe Carenini. ceature-of-ligguisii---ee for improving and interpreting pre-trained language models. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 9456-9469, Singapore, December 2023, Association for Computational Linguistics.
adapters
[19] Xiaoxiao Li, Yuan Zhou, Nicha Dvomek, Muhan Zhang, Siyuan Gao, Juntang Zhuang, Dustin Scheinost, Lawrence H Staib, Pamela Ventola, and James S Duncan. Braingnn: Interpretable brain graph neural network for fmni analysis. Medical Image Analysis, 74:102233, 2021.
[20] Thomas Littlejohns, Jo Holliday, Lorna M Gibson, Steve Garratt, Niels Oesingmann, Fidel Iffaro-Almagno Jimmy D Bell, Chris Boultwood, Rory Collins, Megan C Conroy, al. The uk biobank imaging enhancement of 100,000 participants: rationale, data collection, management and future directions. Nature communications, 11)::2622, 2020.
[21] Charles Lynch, Lucina Q Uddin, Kaustubh Supekar, Amirah Khouzam, Jennifer Phillips, and Vinod Menon. Default mode network in childhood autism: posteromedial cortex heterogeneity and relationship with social deficits. Biological psychiatry, 74(3):212-219, 2013.
[22] Christopher J Markiewicz, Krzysztof J Gorgolewski, Franklin Feingold, Ross Blair, Yaroslav O Halchenko, Eric Miller, Nell Hardcastle, Joe Wexler, Oscar Esteban, Mathias Goncavles, et al. The openneuro resource for sharing of neuroscience data. Elife, 10:471774, 2021.
[23] Josue Ortega Caro, Antonio Henrique Oliveira Fonseca, Christopher Averill, Syed A Rizvi, Matteo Rosati, James L Cross, Prateek Mittal, Emanuele Zappala, Daniel Levine, Rahul M Dhodapkar, et al. Brainlm: A foundation model for brain activity recordings. bioRxiv, pages 2023- 09, 2023.
[24] Anwar Said, Roza Bayrak, Tyler Derr, Mudassir Shabbir, Daniel Moyer, Catie Chang, and Xeno- fon Koutsoukos. Neurograph: Benchmarks for graph machine learning in brain connectomics. Advances in Neural Information Processing Systems, 36:6509-6531, 2023.
[25] Hannah S Savage, Peter CR Mulders, Philip FP Van Fijndhoven, Jasper Van Oort, Indira Tendolkar, Janna N Vrijsen, Christian F Beckmann, and Andre F Marquand. Dissecting task- based fmri activity using normative modelling: an application to the emotional face matching task. Communications Biology, 7(1):888, 2024.
[26] Paul S Scotti, Mihir Tripathy, Cesar Kadir Torrico Villanueva, Reese Kneeland, Tong Chen, Ashutosh Narang, Charan Santhirasegaran, Jonathan Xu, Thomas Naselaris, Kenneth A Norman, et al. Mindeye2: Shared-subject models enable fmri--o-image with 1 hour of data. arXiv preprint arXiv:2403. 12077, 2024.
[27] Qawi K Telesford, Eduardo Gonzalez-Moreira. Ting Xu, Yiwen Tian, Stanley J Colcombe, Jessica Cloud, Brian E Russ, Arnaud Falchier, Maximilian Nentwich, Jens Madsen, et al. An open-access dataset of naturalistic viewing using simultaneous eeg-fmri. Scientific Data, 10(1):554, 2023.
12

---

[28] Nathalie Tzourio-Mazoyee Brigitte Landeau, Dimitri Papathanassiou, Fabrice Crivello, Octave Etard, Nicolas Delcroix, Bernard Mazoyer, and Mare Joliot. Automated anatomical labeling of activations in spm using macroscopic anatomical parcellation of the mni mri single-subject brain. Neuroimage, 15(1):273-28)) 2002.
[29] David C Van Essen, Stephen M Smith, Deanna M Barch, Timothy EJ Behrens, Essa Yacoub, Kamil Ugurbil, Wu-Minn HCP Consortium, et al. The wu-minn human connectome project: an overview. Neuroimage, 80:62-79, 2013.
[30] Jiquan Wang. Sha Zhao, Zhiling Luo, Yangxuan Zhou, Haiteng Jiang, Shijian Li, Tao Li, and Gang Pan. Cbramod: criss-cross brain foundation model for eeg decoding. arXiv preprini arXiv:2412.072366 2024.
[31] Ziquan Wei, Tingting Dan, Jiaqi Ding, and Guorong Wu. Neuropath: A neural pathway transformer for joining the dots of human connectomes. The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024.
[32] Michael W Weiner, Dallas P Veitch, Paul S Aisen, Laurel A Beckett, Nigel Cairns, Jesse Cedarbaum, Michael C Donohue, Robert C Green, Danielle Harvey, Clifford R Jack Jr, et al. Impact of the alzheimer's disease neuroimaging initiative, 2004 to 2014. Alzheimer's 's & Dementia, 11(7):865 884, 2015.
[33] Jiaxing Xu, Yunhan Yang, David Huang, Sophi Shilpa Gururajapathy, Yiping Ke, Miao Qiao, Alan Wang, Haribalan Kumar, Josh McGeown, and Eryn Kwon. Data-driven network neu- roscience: On data collection and benchmark. Advances in Neural Information Processing Systems, 36:21841-218566 2023.
[34] Yanwu Yang, Chenfei Ye, Guinan Su, Ziyao Zhang, Zhikai Chang, Hairui Chen, Piu Chan, Yue Yu, and Ting Ma. Brainmass: Advancing brain network analysis for diagnosis with large-scale self- sulevvuuer learning. IEEE Transactions on Medical Imaging, 2024.
[35] Jiazuo Yu, Yunzhi Zhuge, Lu Zhang, Ping Hu, Dong Wang. Huchuan Lu, and You He. Boosting continual learning of vision-language models via mixture-of-experts adapters. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23219-23230, 2024.
[36] Shu Zhang, Xiang Li, Jinglei Lv, Xi Jiang, Lei Guo, and Tianming Liu. Characterizing and differentiating task-based and resting state fmri signals via two-stage sparse representations. Brain imaging and behavior, 10:21-32. 2016.
[37] Zhuofan Zong, Bingqi Ma, Dazhong Shen, Guanglu Song, Hao Shao, Dongzhi Jiang, Hongsheng Li and Yu Liu. Mova: Adapting mixture of vision experts to multimodal context, arXiv preprint arXiv:2404. 3346, 2024.
13

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: See Sec 1
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: See Sec
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA].
14

---

Justification: [NA]
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes)
Justification: See Sec 4 and Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reprodueibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
15

---

Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ cppppi.ccppulli//uuuii wwwiiicgguie//eeeeeeeeeis CoodeSubeiiie ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: See Sec
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: See Sec 4
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
16

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g.. preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: httpp://///uriss..//// http///urrs.cccppui////e http://c.....p.iittttttccccdeeed
Answer: [Yes]
Justification: [Yes]
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is a special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to
17

---

generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: See Sec
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, wwppppswpptseeee whworee.wthhhe....c.. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's is creators.
13. New assets
Question: Are new assets introduced in the paper well documented and the documentation provided alongside the assets?
18

---

Answer: [Yes]
Justification: See Appendix
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least the minimum wage in the country of the data collector.
Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: [NA]
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
19

---

Justification: [NA]
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy htteas////eerrpppoccc///o/rreers for what should or should not be described.
20

---

A Accessibility
Public data is accessible via internet (UKH2 HCPA HCPYA ADN PPMI, ABIDE, and Taowu can be found here The licenses to obtain those data can also be accessed on the websites. The codes and data split settings can be acquired via this code repository"
Data preprocessing
The neuroimage processing used for ADNI, UKB, HCPYA, and HCPA consists of the following major steps: (1) We segment the weeehhhte image into white matter, gray matter, and cerebral spinal fluid using FSL software [T6]. (2) On top of the tissue segmentation in Fig. / we parcellate the cortical surface of fMRI into cortical regions according to the atlas as a regional signal of time-series Fig. where FC, the end, the Pearson correlation coefficient between regional time-series.
Brain extraction, tissue segmentation, aay brain mask refinement,...
T1w MRI
matrix
fMRI
Head-mation estimation, slice time corraction, ... confounds estimation,
Figure 7: General workflows for processing T1-weighted image (Tlw MRI) and functional MRI (fMRI). The output is shown the right, including the brain network of FC.
C Computing environments and hyperparameters
The experiments done Linux system with NVIDIA RTX 6000 Ada. Batch size and learning rate are set as 128 and le-4, respectively. The maximum epoch is set as 200 and Chid 2048. Training will be early stopped if accuracy keeps dropping in 50 epochs.
D
Comparison between previous works
We list the comparison of experimental datasets between previous works in Table 7
E
Computational complexity
The limitation of BrainMoE is more computational complexity, as listed in Table 8 BrainMoE with 12 experts spends 4x more time than basclines. All-in-one with 36 experts nearly doubles the time cost.
https: httpp::://ww..ukbobban... uk/
.rg
https://adni:s Loni wwt.aai.uucceeee. usc.edu/
21

---

Table 7: The comparison of experimental datasets between previous works.
BrainLM (2024) [33
BrainMass (2024) [34
BrainJEPA (2024)
I BrainMoE (Ours)
Brain atlas
424
C200
Schaefer400
AAL116
resting, 11 types of tasking
Cognitive state
resting, task-hariri
resting
resting
UKB, HCP, OpenNeuron
Pre-train dataset
UKB, HCP
UKB, HCP
UKB, HCP
Pre-train
61,038
64,584
40,162
68,251
ASD, ADHD. AD, PD. MDD 34M
HCP, ASD, AD, PD, SZ 709M
Fine-tune dataset
UKB, HCP
UKB, HCP, ADNI
Parameter amount
650M
307M
Table 8: Computational time cost of BrainMoE inference with two existing architectures and the all all-in-one BrainMoE on the ABIDE dataset.
Test time (ms/sample)
BrainMass
BrainJEPA
Single model
37.08 157.60
28.13
BrainMoE
133.26
All-in-one
287.21
F Visual decoding potential
Visual decoding task for a new dataset NSD [2] has also been evaluated for MindEye2 (26] as the baseline and BrainMoE. We pre-trained MindEye2s as the specific experts for long-term (novel trials) and short-term memory (easy/hard trials), respectively. Since visual decoding is generative task, output contains much higher dimensions (256x 1664 vs. class number to than downstream tasks focused in the main text. Therefore, we skipped our cogmition adapter by weighted summing the diffusion prior of two experts with the BrainMoE routing probabilities. Both baseline and BrainMoE are pretrained with subjects 2-7 and finetuned with subject 1 on the entire 40 sessions. The final train and test losses, cosine similarity, and Mean Squared Error (MSE) during finetuning are listed Table g Given the evidence that the performance of BrainMoE is better than the single expert MindFye2, there is potential for BrainMoE to expand to visual decoding.
Table 9: Visual decoding performance.
MindEye2
BrainMoE
Train loss
9.639
7.994
Test loss 11.142
9.405
Cos. Sim.
0.778
0.840
MSE
0.301
0.261
G
Scalability analysis
MLP as a universal predictive head is used in related works for the MoE adapter. Fig. 8 is the comparison between MLP and the proposed cognition adapter with different amounts of learnable parameters.
H
Visualization
The attention weights conducted by BrainMoE with different is visualized in Fig. g can observe: (1) Advanced by the cognition adaptor, BrainMolE agrees with current neuroscience knowledge since mainly attends DAN and DMN for ASD 12 21 SMN and FPN for PD [7]. (2) Differences are slight across enlarged k, indicating that the router produces consistent expert weights.
22

---

abide
ppmi
adni
0.5
0.8
0.3
0.4
Our adapter
0.6
0.2
MLP adapter
0.2
0.1
0.5
0.5
param # Figure 8: The scalability of the MLP adapter and our adapter on disease prediction, where the y-axis the fine-tuning loss.
param e
param
k=1
k=50%
k= 100%
Superior
Left
Superior Left
Superio
Left
ASD
Superior
Left
Supero
Superior
Left
PD
Default Mode
Visual
Frontoparietal
Limbic
Ventral attention
Sensorimotor
Dorsal attention
Figure 9: Visualization of attention weights by FC reconstruction BrainMoF.
23


Reviews:
[{"review_id": "s6Zb2yAP8i", "reviewer": "Reviewer_ZhRL", "summary": "This paper introduces a MoE-based foundation model for brain modeling. BrainMoE pre-trains expert models specific to different cognitive states, aggregates their embeddings using a Transformer-based cognition adapter, and demonstrates improved downstream performance on disease diagnosis, behavior, and sex classification across seven datasets.", "questions": "1. How is the number of experts chosen?\n2. MoE can be prone to suffer from the unbalanced issue, i.e., only a few experts are activated and trained. Does BrainMoE has any regularization to address this?\n3. Can you provide further analysis on which experts (cognitive tasks) are most relevant for specific downstream tasks, and whether these align with neuroscientific knowledge?", "limitations": "I did not see the discussion on limitations", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strength:\n1. BrainMoE considers the task heterogeneity and pretrain dedicated experts for different cognitive states. \n2. The approach is modular—any model pre-trained on a given cognitive state can be used as an expert. The cognition adapter further allows flexible downstream adaptation without restriction to a particular input type (BOLD or FC).\n3. BrainMoE is tested across seven varied datasets from three different preprocessing pipelines, spanning clinical populations (e.g., Autism, Parkinson’s, Alzheimer's, Schizophrenia) and behavioral labeling, offeringh a comprehensive evaluation.\n\nWeakness:\n1. The results presented in Tables 1 and 2 are strong, but some standard baselines are omitted—for example, several datasets appear not to include a naive fine-tuning of a single (non-stratified) expert per cognitive state, or late fusion of independent predictions. \n2. While Figure 5 highlights neurobiological alignment in attention maps, interpretability stops at the task-level. Analysis of which tasks or experts are favored for which downstream applications, and why certain combinations work better, would inform neuroscientific implications and practical use.\n3. As discussed in Table 5, inference cost for BrainMoE is substantially higher than baselines (4–10x), which could limit large-scale or real-time applications, and there is limited discussion on possible amortized efficiency gains or pruning.", "was_revised": true, "final_justification": "This paper is methodologically sound, with comprehensive experiments demonstrating significant performance gains. In the rebuttal phase, the authors adequately addressed most of my prior questions and concerns, including clarifications on the experiments and ablations.\n\nHowever, the core idea of training independent experts for each cognitive state feels relatively straightforward and akin to a specialized ensemble method, while the cognition adapter primarily functions as a trainable classification head (via task-query embeddings) during fine-tuning. Overall, while the method is robust and well-executed, its novelty is somewhat limited, as it largely adapts existing Mixture-of-Experts concepts from other domains without groundbreaking innovations in the neuroimaging context.\n\nFurthermore, given the relevance of cognitive states to diseases like Alzheimer's (AD), the paper could better relate individual experts to specific disease mechanisms and provide more in-depth neuroscientific analyses. For instance, while attention visualizations align with known brain networks (e.g., DMN for Autism, SMN for Parkinson's), deeper expert-level interpretations linking stratified cognitive embeddings to clinical outcomes would strengthen the contributions. This should be straightforward, as the main motivation of the paper is to develop separate experts for each cognitive state.\n\nGiven the above reasons, I would maintain my score as 3.", "rebuttal": "## Brief\n\nThank you for your insightful comments and questions. The main concern of your reviews stands on the lack of evaluation per individual expert and late fusion. We have demonstrated all evaluations in our rebuttal. However, the evaluation per individual expert is lengthy and uncommon in previous brain modeling and MoE studies. We will keep them brief in the final version.\n\n## **W1**\n\nSince resting-state fMRI was mainly used in previous brain foundation models (see L.40 in the manuscript), we included only resting-state experts as baselines. Here, we evaluated the standard baselines, a single expert per tasking-state, and reran the experiments of Tables 1 and 2 in the manuscript. Tables A and B show results for BrainMass-MLP baseline. \n\n**Baseline results** show that there are consistently existing task-specific experts (e.g., language for AD, working memory for PD) outperforming the resting-state baselines (30k version in the manuscript), confirming that task-state fMRI contains valuable information for brain modeling. Conclusively, BrainMoE still holds the best performance compared to all datasets and baselines. This supports that the utilization of cognitive embeddings from BrainMoE leads to more robustness of brain modeling than naively training a single task of fMRI. Furthermore, the expert combinations learned by BrainMoE and their strong relevance to downstream applications, shown in W2 response, such as Language for ASD, and Working memory for PD, offer valuable neuroscientific insights that current baseline models fail to capture.\n\n**The late fusion** of independent experts is also tested across 7 datasets. Please refer the results to Table A for Reviewer PTHj. The superior performance of BrainMoE and the contrast of selected expert routing patterns between BrainMoE and the late fusion provide neuroscientific insight and support our methodological novelty.\n\n---\n *Table A*\n\n|Expert|ADNI (AD)|ABIDE (ASD)|PPMI (PD)|Taowu (PD)|SZ (SZ)|HCPA (3-task)|HCPYA (7-task)|\n|-|-|-|-|-|-|-|-|\n|Hariri (n=24,672)|75.32$_{\\\\pm7.06}$|65.41$_{\\\\pm3.85}$|59.78$_{\\\\pm13.93}$|64.33$_{\\\\pm18.39}$|78.09$_{\\\\pm9.20}$|**89.42$_{\\\\pm0.64}$**|**77.58$_{\\\\pm2.35}$**|\n|Working memory (n=1,756)|76.15$_{\\\\pm6.37}$|61.31$_{\\\\pm3.71}$|**60.86$_{\\\\pm13.68}$**|**79.86$_{\\\\pm14.51}$**|78.34$_{\\\\pm9.30}$|80.53$_{\\\\pm0.53}$|64.59$_{\\\\pm1.05}$|\n|Motor (n=1,712)|76.35$_{\\\\pm6.26}$|60.30$_{\\\\pm4.37}$|57.86$_{\\\\pm11.09}$|72.57$_{\\\\pm26.98}$|**81.04$_{\\\\pm8.95}$**|78.74$_{\\\\pm0.71}$|60.41$_{\\\\pm1.93}$|\n|Gambling (n=1,705)|76.35$_{\\\\pm6.26}$|64.18$_{\\\\pm4.50}$|58.17$_{\\\\pm13.76}$|79.24$_{\\\\pm21.92}$|79.12$_{\\\\pm8.05}$|80.98$_{\\\\pm0.44}$|65.74$_{\\\\pm2.52}$|\n|Facename (n=1,699)|75.32$_{\\\\pm7.06}$|51.34$_{\\\\pm4.73}$|59.24$_{\\\\pm13.42}$|65.19$_{\\\\pm24.41}$|76.95$_{\\\\pm9.01}$|70.40$_{\\\\pm0.49}$|28.05$_{\\\\pm1.09}$|\n|Language (n=1,674)|**78.30$_{\\\\pm8.99}$**|**65.83$_{\\\\pm2.35}$**|60.84$_{\\\\pm14.54}$|64.43$_{\\\\pm28.40}$|80.63$_{\\\\pm8.30}$|78.73$_{\\\\pm0.69}$|58.08$_{\\\\pm1.00}$|\n|Social (n=1,674)|75.32$_{\\\\pm7.06}$|64.34$_{\\\\pm3.62}$|60.59$_{\\\\pm12.91}$|67.24$_{\\\\pm24.05}$|77.88$_{\\\\pm9.76}$|81.00$_{\\\\pm0.84}$|66.90$_{\\\\pm1.69}$|\n|Relational (n=1,609)|77.89$_{\\\\pm6.23}$|61.13$_{\\\\pm6.14}$|55.66$_{\\\\pm15.59}$|76.33$_{\\\\pm17.85}$|79.37$_{\\\\pm8.36}$|74.96$_{\\\\pm1.19}$|52.82$_{\\\\pm1.61}$|\n|Emotion (n=1,478)|76.52$_{\\\\pm6.18}$|60.29$_{\\\\pm3.83}$|58.46$_{\\\\pm15.24}$|69.67$_{\\\\pm22.97}$|78.01$_{\\\\pm8.71}$|85.01$_{\\\\pm0.90}$|70.69$_{\\\\pm1.55}$|\n|Carit (n= 698)|77.87$_{\\\\pm5.79}$|63.01$_{\\\\pm2.32}$|59.81$_{\\\\pm15.31}$|70.10$_{\\\\pm15.38}$|79.02$_{\\\\pm9.12}$|77.64$_{\\\\pm1.39}$|55.04$_{\\\\pm1.70}$|\n|Vismotor (n=637)|75.32$_{\\\\pm7.06}$|63.70$_{\\\\pm3.70}$|58.45$_{\\\\pm13.84}$|73.43$_{\\\\pm12.58}$|76.95$_{\\\\pm9.01}$|56.10$_{\\\\pm0.41}$|27.82$_{\\\\pm0.51}$|\n\n---\n *Table B*\n\n|Expert|ADNI (sex)|ABIDE (sex)|PPMI (sex)|Taowu (sex)|HCPA (sex)|HCPYA (sex)|\n|-|-|-|-|-|-|-|\n|Hariri (n=24,672)|56.25$_{\\\\pm9.06}$|75.13$_{\\\\pm4.97}$|**67.92$_{\\\\pm11.43}$**|67.86$_{\\\\pm27.70}$|**65.44$_{\\\\pm0.70}$**|**64.01$_{\\\\pm4.18}$**|\n|Working memory (n=1,756)|50.62$_{\\\\pm5.93}$|**75.92$_{\\\\pm4.43}$**|58.90$_{\\\\pm11.28}$|62.43$_{\\\\pm23.44}$|58.73$_{\\\\pm2.54}$|58.77$_{\\\\pm0.90}$|\n|Motor (n=1,712)|55.78$_{\\\\pm7.14}$|74.99$_{\\\\pm5.43}$|60.03$_{\\\\pm10.91}$|62.86$_{\\\\pm24.85}$|59.28$_{\\\\pm1.95}$|54.62$_{\\\\pm5.77}$|\n|Gambling (n=1,705)|61.17$_{\\\\pm7.01}$|74.76$_{\\\\pm5.56}$|63.37$_{\\\\pm8.16}$|70.00$_{\\\\pm28.67}$|63.52$_{\\\\pm1.05}$|61.22$_{\\\\pm1.12}$|\n|Facename (n=1,699)|65.61$_{\\\\pm7.95}$|73.87$_{\\\\pm5.66}$|54.65$_{\\\\pm7.45}$|52.10$_{\\\\pm26.55}$|58.36$_{\\\\pm2.47}$|59.69$_{\\\\pm2.97}$|\n|Language (n=1,674)|63.02$_{\\\\pm7.78}$|75.42$_{\\\\pm5.23}$|55.95$_{\\\\pm9.09}$|57.19$_{\\\\pm23.50}$|60.37$_{\\\\pm1.61}$|58.90$_{\\\\pm2.83}$|\n|Social (n=1,674)|60.09$_{\\\\pm5.19}$|74.55$_{\\\\pm5.04}$|62.87$_{\\\\pm10.80}$|69.05$_{\\\\pm26.14}$|64.30$_{\\\\pm1.55}$|59.91$_{\\\\pm2.28}$|\n|Relational (n=1,609)|**65.72$_{\\\\pm9.28}$**|75.67$_{\\\\pm6.28}$|55.51$_{\\\\pm9.74}$|**74.00$_{\\\\pm23.65}$**|57.84$_{\\\\pm1.72}$|57.32$_{\\\\pm3.33}$|\n|Emotion (n=1,478)|61.64$_{\\\\pm10.77}$|75.11$_{\\\\pm5.26}$|63.26$_{\\\\pm8.96}$|61.10$_{\\\\pm29.22}$|64.91$_{\\\\pm1.66}$|62.29$_{\\\\pm3.20}$|\n|Carit (n= 698)|56.31$_{\\\\pm17.07}$|75.36$_{\\\\pm4.96}$|63.23$_{\\\\pm9.37}$|58.86$_{\\\\pm24.63}$|59.75$_{\\\\pm1.06}$|59.23$_{\\\\pm2.81}$|\n|Vismotor (n=637)|63.91$_{\\\\pm5.26}$|73.87$_{\\\\pm5.66}$|64.04$_{\\\\pm8.85}$|62.10$_{\\\\pm23.25}$|46.42$_{\\\\pm1.76}$|46.65$_{\\\\pm5.00}$|\n\n\n## **W2, Q3**\n\nHere, we collect the expert routing probabilities of BrainMoE on clinical datasets and count the combination of top experts (top 1 if the probability > 0.9, otherwise top 2). We can observe the most frequent combination (Table C for reviewer PTHj):\n - AD: [Working memory] 31.1%, [Rest, Motor] 20%\n - ASD: [Language, Relational (relationships between stimuli)] 14.9%, [Rest, Relational] 8.3%\n - PD: [Language, Relational] 20%, [Vismotor (color change and button press), Relational] 9.1%\n - SZ: [Gambling, Relational] 18.5%, [Facename (face and name recall)] 14.3%\n\nThe relatively more frequent activation of these expert combinations, e.g., [Language, Relational] for ASD and PD, suggests that BrainMoE captures complementary neural patterns that individual experts might miss.\n\nOn the other hand, combinations for different applications align with neuroscientific knowledge: \n - For AD, the dominance of [Working memory] aligns with that AD is commonly attributed to central executive impairment and frontal lobe dysfunction [1].\n - For ASD, [Language, Relational] mirrors impairments in relational memory processing, where individuals with ASD show difficulties in forming relations between items and encoding relational but not item information [2], along with the language network showing atypical functional connectivity [3].\n - For PD, [Vismotor, Relational] reflects the visuomotor processing deficits, including impaired decision-making cascades and altered cortical sensorimotor processing [4].\n - For SZ, those combinations reflect deficits in reward learning and decision-making processes, as well as impaired facial processing and recognition abilities that are characteristic of the disorder [5].\n\n[1] 10.1016/j.cortex.2010.12.002\n\n[2] 10.1002/aur.1493\n\n[3] 10.1002/aur.2171\n\n[4] 10.3390/brainsci13081173\n\n[5] 10.3389/fneur.2019.00990\n\n## **W3**\n\nBrainMoE outperformed all baselines from the largest (n=4,863) to the smallest dataset (n=40), demonstrating the superior robustness. On the other hand, the running time of BrainMoE has fulfilled the real-time requirement for fMRI applications, which takes a much longer time (10-15mins) than BrainMoE (133-287ms) for data acquisition using a common 3T scanner [6].\n\nThe possible amortized efficiency gains can be the pre-computation of cognitive embeddings. In fact, our experiments for BrainMass experts (FLOPs: 26.96M) have the cognitive embeddings computed on-the-fly, while BrainJEPA (FLOPs: 5.32T) computes them prior to the finetuning. Since the cognitive embeddings are frozen and fixed per data, pre-computation of the complex experts, BrainJEPA in our case, can save the time of iterative computation during BrainMoE finetuning. This trick enhanced efficiency as listed in Table 5, where BrainJEPA with the trillion-level FLOPs costs less time than BrainMass with the million level FLOPs. We will include the discussion of this possible amortized efficiency pruning in the “Limitation” section of the final version.\n\n[6] 10.3174/ajnr.A8067\n\n## **Q1**\n\nThe number of experts (N) equals $N_{arch}\\times N_{cog}$, where $N_{arch}$ is the number of available model architectures and $N_{cog}$ is the number of distinct cognitive states. E.g., there are three types of architecture (BrainMass, BrainJEPA, and Cog. Classif.) implemented in our BrainMoE All-in-one, so that $3\\times 12=36$. We will revise the description in L.113-L.120 to clarify the expert number.\n\n## **Q2**\n\nYes, the design of BrainMoE cognition adapter can address the unbalanced expert. \n\nAs shown in Fig. 4 (b) and Eq. (3), the final prediction of BrainMoE is not the explicit fusion of expert embeddings but the transformation of input FC or BOLD weighted by the cross-attention between expert embeddings. This implicit usage of experts avoids their overfitted or underfitted cognition embeddings to directly bias the output of BrainMoE. Thus, it addressed unbalanced brain experts supported by empirical evidence: \n\nThe pattern comparison (Table B vs. Table C for Reviewer PTHj) between the router trained by our BrainMoE approach and the generic late fusion MoE. Late fusion favors the top-activated experts rather than the domain experts across all datasets. As in Table B for Reviewer PTHj, the top combination among 7 datasets consistently contains Rest (n=29,971), Hariri (n=24,672), and Relational (n=1,609) experts. In contrast, as in Table C for Reviewer PTHj, BrainMoE has diverse top combinations for heterogeneous downstream tasks and similar top combinations for homogeneous applications, aligning with neuroscientific knowledge"}, {"review_id": "Zwt7p1LIV6", "reviewer": "Reviewer_PTHj", "summary": "This paper introduces a framework for fMRI understanding foundation model, called BrainMoE. It first trains each expert using fMRI data from corresponding datasets and then fine-tunes the fMRI representations by training a router and a query-Transformer. The authors conducted a number of experiments on a categorization task to evaluate their BrainMoE.", "questions": "After carefully reading the method section, I still have the following questions:\n\n+ There should be significant differences among the datasets, such as variations in the dimensions of the BOLD signals. How does BrainMoE address this issue? I don’t seem to have found any explanation regarding this in the paper.\n\n+ The authors introduced a binary supervision signal during pretraining to indicate whether the data is from resting-state fMRI. What is the benefit of this approach in the context of pretraining?\n\n+ In Figure 3(b), the representations used for downstream tasks appear to come from a combination of multiple brain experts, whereas in Figure 4(a), the representations seem to be derived from any single model. This is somewhat unclear.", "limitations": "I believe one of the main limitations of the paper lies in its experimental evaluation. BrainMoE is a robust fMRI representation model, and evaluating it solely on classification tasks is, in my opinion, insufficient. Assessing its performance on other types of tasks would significantly enhance the value of the paper — for example, tasks like visual decoding [1] (i.e., reconstructing visual stimuli [2] from fMRI signals).\n\nThe other minor limitation is the writing, figures, and formatting of the paper need further improvement. The expression in the introduction and methods sections requires additional polishing. The figures should be more aesthetically pleasing and concise, and the formatting needs further revision (for example, part of the text is obscured in Figure 3).\n\n[1] Emily J. Allen et al. A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence. Nature neuroscience.\n\n[2] Paul S. Scotti et al. Mindeye2: Shared-subject models enable fmri-to-image with 1 hour of data. ICML 2024.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 3, "strengths_and_weaknesses": "The strength of this paper lies in its massive experimental evaluation, which includes three pretraining datasets and seven downstream task datasets. However, its weaknesses include the lack of insight or novelty in the BrianMoE approach, the issue of homogeneity among the downstream tasks used for evaluation, and the overall clarity of the writing, which I believe needs further revision for better expression.", "was_revised": true, "final_justification": "The authors’ response has addressed all of my concerns. Their explanations clearly demonstrate the novelty of the work. The training results on the decoding task indicate that BrainMoE is likely adaptable to this task. So, I am raising my rating.", "rebuttal": "## Brief\n\nThank you for your insightful comments and questions. The main concern of your reviews is derived from the misunderstanding of our methodological novelty and the lack of evaluations other than classification. We added results from a generic MoE to show the novelty of our BrainMoE approach, as well as regressive and generative applications.\n\n## **W1**\n\nAlong with the agreement of our novelty by reviewer Yxzq, one key novelty is innovating the BrainMoE approach that can significantly enhance the robustness of brain foundation model by pre-training with tasking-state fMRI, which was not fully explored in previous studies. Due to various sample sizes of pre-training datasets (Table A for Reviewer ZhRL), the data unbalance issues of MoE are amplified in the brain modeling area, as indicated by the performance of a generic MoE framework, the late fusion MoE [4], in Table A. To address this limitation, our BrainMoE presents innovative learning strategies for our new cognitive MoE adapter, to tackle the data unbalance issue. \n\n**Effectiveness of BrainMoE**: We can see late fusion MoE can sometimes outperform the baseline, but it is consistently worse than BrainMoE except for the smallest dataset Taowu (n=40). Unlike BrainMoE has the cognition adapter to implicitly utilize expert embeddings, the late fusion explicitly combines expert predictions, therefore cannot handle the unbalance issue.\n\n---\n*Table A*\n\n||ADNI|ABIDE|PPMI|Taowu|SZ|HCPA|HCPYA|\n|-|-|-|-|-|-|-|-|\n|Baseline|80.70$_{\\pm7.85}$|67.81$_{\\pm3.91}$|59.77$_{\\pm14.22}$|68.00$_{\\pm21.46}$|77.63$_{\\pm8.56}$|90.63$_{\\pm0.74}$|81.27$_{\\pm1.27}$|\n|Late fusion MoE|73.33$_{\\pm10.87}$|69.89$_{\\pm306}$|61.11$_{\\pm15.29}$|**91.24$_{\\pm11.72}$**|76.95$_{\\pm9.50}$|94.96$_{\\pm2.64}$|88.58$_{\\pm3.39}$|\n|BrainMoE|**81.23$_{\\pm11.00}$**|**70.26$_{\\pm3.40}$**|**62.97$_{\\pm13.94}$**|88.57$_{\\pm12.51}$|**83.10$_{\\pm11.33}$**|**96.67$_{\\pm0.77}$**|**93.19$_{\\pm0.72}$**|\n\n\nRegarding top expert combinations (top 1 if the probability > 0.9, otherwise top 2) chosen by the router, it favors the top activated experts rather than the domain experts across all datasets. \n\nAs listed in Table B, the combination from late fusion consistently contains Rest (n=29,971), Hariri (n=24,672), and Relational (n=1,609) experts. In contrast, BrainMoE has diverse and similar top combinations for heterogeneous and homogeneous applications, respectively (Table C). The contrast of selected combinatory patterns between routers trained by BrainMoE approach and the late fusion provides neuroscientific insight and supports our methodological novelty.\n\n\n---\n*Table B*\n\n|(%)|ADNI|ABIDE|PPMI|Taowu|SZ|HCPA|HCPYA|\n|-|-|-|-|-|-|-|-|\n|Carit|0|0|0|0|0|5|0|\n|Emotion|0|4.6|1.4|0|0|2.8|0|\n|Emotion-Hariri|0|0|4.3|0|0|0|0|\n|Emotion-Motor|0|0|0|7.5|0|0|0|\n|Facename|18.5|4.4|0|0|0|2.8|0|\n|Facename-Relational|0|0|3.3|0|0|0|0|\n|Gambling|0|5.3|1|5|9.5|14|0|\n|Gambling-Motor|0|0|5.3|2.5|0|0|0|\n|Gambling-Relational|0|0|0.5|5|0|0.2|0|\n|Hariri|8.1|2.1|10.5|10|**47.1**|**16.3**|**27.1**|\n|Language|0|8.1|9.6|7.5|0|0|2.8|\n|Language-Relational|0|10.9|0|2.5|0|0|0|\n|Motor|0|0|0|7.5|0|4.8|4.4|\n|Motor-Hariri|0|0|0|5|0|0.2|0.5|\n|Relational|**20**|**15.2**|**22**|**25**|10.1|28|**22.6**|\n|Rest|**45.9**|10|**12.4**|0|**30.2**|2.5|20|\n|Rest-Emotion|0|0|5.7|0|0|0|0|\n|Rest-Hariri|5.9|0|4.8|0|0|0.1|1.6|\n|Rest-Motor|0|0|4.3|2.5|0|0|1.5|\n|Rest-Relational|0|**18.8**|5.7|0|0|0.3|0.5|\n|Rest-WM|0|0|0|5|0|0|0.2|\n|Social|0|0|0|2.5|0|5.3|0|\n|Social-Relational|0|0|0|5|0|0.2|0|\n|WM|0|8.2|0|2.5|0.5|**14.2**|7.9|\n|WM-Hariri|0|4.4|0|0|1.1|0|0.3|\n|WM-Motor|0|0|9.1|0|0|0|0|\n|WM-Relational|0|0|0|0|0|0.2|6.6|\n\n\n---\n*Table C*\n\n|(%)|ADNI|ABIDE|PPMI|Taowu|SZ|HCPA|HCPYA|\n|-|-|-|-|-|-|-|-|\n|Carit-Facename|0.0|7.7|0.0|0.0|0.0|0.0|0.0|\n|Emotion-WM|0.0|0.0|3.8|10.0|0.0|0.0|0.4|\n|Facename|0.0|1.9|0.5|5.0|**14.3**|4.9|2.2|\n|Facename-Relational|0.0|0.0|1.9|2.5|0.0|4.4|0.1|\n|Gambling|7.4|3.9|0.5|5.0|0.0|2.2|**9.4**|\n|Gambling-Relational|0.0|5.9|0.0|0.0|**18.5**|0.0|3.6|\n|Hariri|0.0|0.0|2.9|0.0|3.7|10.7|**11.8**|\n|Language|0.7|3.9|5.3|**15.0**|0.0|**13.0**|9.0|\n|Language-Emotion|0.0|0.2|5.7|0.0|0.0|0.1|0.1|\n|Language-Facename|0.0|7.0|0.0|0.0|3.7|0.1|0.3|\n|Language-Motor|0.0|0.0|4.3|0.0|0.0|0.0|0.2|\n|Language-Relational|0.0|**14.9**|5.7|**20.0**|0.0|0.3|1.5|\n|Language-Rest|8.1|0.0|4.8|5.0|0.0|1.6|1.4|\n|Motor|0.0|0.2|0.0|0.0|0.0|11.5|2.0|\n|Motor-Relational|0.0|7.3|0.0|10.0|0.0|0.7|0.6|\n|Relational|0.0|3.9|4.3|12.5|14.3|**16.4**|7.3|\n|Rest|3.0|1.6|7.2|2.5|7.9|11.3|7.2|\n|Rest-Gambling|0.0|0.0|0.0|0.0|0.0|0.2|6.5|\n|Rest-Hariri|6.7|0.0|0.5|0.0|0.0|1.9|0.5|\n|Rest-Motor|**20.0**|0.0|1.0|0.0|5.8|0.0|0.1|\n|Rest-Relational|0.0|**8.3**|1.9|0.0|0.0|0.2|1.8|\n|Rest-Social|0.0|0.0|1.0|10.0|9.5|0.0|0.0|\n|Social|0.0|0.0|1.0|0.0|0.5|4.5|0.0|\n|Vismotor-Carit|0.0|6.1|0.0|0.0|0.0|0.0|0.0|\n|Vismotor-Relational|0.0|0.0|**9.1**|0.0|0.0|0.2|0.0|\n|Vismotor-Rest|0.0|0.0|0.0|0.0|6.3|0.1|0.0|\n|Vismotor-WM|8.9|0.0|0.0|0.0|0.0|0.0|0.0|\n|WM|**31.1**|3.3|0.0|0.0|0.5|2.2|6.7|\n|WM-Facename|0.0|6.7|0.0|0.0|0.0|0.0|0.0|\n|WM-Gambling|12.6|0.9|0.0|0.0|0.0|0.3|0.1|\n|WM-Hariri|0.0|0.0|7.2|0.0|6.3|0.0|1.2|\n|WM-Motor|0.0|0.4|**9.1**|0.0|0.0|0.0|0.1|\n|WM-Relational|0.0|6.3|0.0|0.0|3.7|1.3|3.4|\n|WM-Social|0.0|0.0|5.3|0.0|0.0|0.0|0.0|\n\n## **W2**\n\nThe challenge of heterogeneity from downstream tasks among our 7 datasets is as critical as previous brain foundation models (Appendix Table 6). The number of different downstream datasets is the highest, i.e., 5 cohorts, compared to the most recent brain foundation models. In addition, downstream tasks are also more variable than others since there are four types of brain disorders (AD, ASD, PD, SZ) and two types of non-clinical tasks (sex, cognitive state). We will add the variations of our downstream tasks to Table 6.\n\n## **W3, Lim.2**\n\nWe appreciate your suggestion, and we are committed to thoroughly revising the manuscript. Specifically, we will highlight the innovation of BrainMoE and provide more comprehensive supporting results as shown in this rebuttal.\n\n## **Q1**\n\nBOLD signal is framed in a spatial-temporal data structure. One novelty of BrainMoE is modeling the brain with arbitrary input type (see L.68 point #2). Input of brain experts is denoted by **X** (L.106), which has variable temporal dimensions. Arbitrary brain expert maps **X** to **Z** (L.137) to have the same dimension $C_{hid}$ for BrainMoE modules. Although in practice, input shapes $C_{in}=M$ for BrainMass experts and $C_{in}=T$ for BrainJEPA experts are different, their cognitive embeddings both have the same shape $C_{hid}$. Therefore, BrainMoE can work with various input shapes. We will revise the method section with some examples of input shape to smooth the description of BrainMoE handling variable input shape.\n\n## **Q2**\n\nThe motivation behind this is considering the variations of brain expert architecture to the extent of supervision type. The utilization of binary supervision complements the gap between standard brain classification models, e.g., transformer-based [3], and the brain foundation models. Empirical evidence in our work shows that binary supervision can benefit the performance of BrainMoE on clinical (Cog. Classif. in Table 3) and non-clinical applications (All-in-one in Table 4), indicating the potential of standard brain classification architectures for brain foundation modeling. We will clarify this motivation in the Introduction section and discuss the results in the Conclusion section.\n\n## **Q3**\n\nWe illustrated the overall framework of BrainMoE and the learning strategy in Fig 3, including pre-training and fine-tuning. Fig 4 illustrates the novel blocks in our framework, particularly the brain expert and the cognition adapter. Fig 4a shows the pre-training of brain experts corresponding to Fig 3a instead of b, the downstream tasks are therefore not involved in this step. On the other hand, Fig 4b shows the fine-tuning for downstream, and it corresponds to Fig 3b. We will combine Fig 3 and Fig 4 as one figure to make the method description clear.\n\n\n## **Lim.1**\n\nAge regression for 6 datasets has been evaluated for the best baseline, 68k version of BrainMass, and BrainMoE. We can see the performance of BrainMoE is the best. The improvement is especially significant for ABIDE (n=1,025, 6-58 yrs) with MSE 36.77 -> 4.86. This empirical evidence further supports the robustness of BrainMoE.\n\n---\n*Table D*\n\n|MSE (unit: yr)|ADNI (age)|ABIDE (age)|PPMI (age)|Taowu (age)|HCPA (age)|HCPYA (age)|\n|-|-|-|-|-|-|-|\n|BrainMass-68k|36.28$_{\\pm19.83}$|36.77$_{\\pm17.21}$|33.09$_{\\pm21.87}$|38.10$_{\\pm28.33}$|22.66$_{\\pm7.51}$|5.46$_{\\pm2.69}$|\n|BrainMoE|**36.27$_{\\pm 10.23}$**|**4.86$_{\\pm2.67}$**|**29.89$_{\\pm8.39}$**|**30.72$_{\\pm12.25}$**|**10.56$_{\\pm2.86}$**|**3.45$_{\\pm1.08}$**|\n\nVisual decoding task for a new dataset NSD [1] has also been evaluated for MindEye2 [2] as the baseline and BrainMoE. We pre-trained two MindEye2s as the specific experts for long-term (novel trials) and short-term memory (easy/hard trials), respectively. Since visual decoding is a generative task, output contains much higher dimensions (256x1664 vs. class number 2-7) than downstream tasks focused on our study. Therefore, we skipped our cognition adapter by weighted summing the diffusion prior of two experts with the BrainMoE routing probabilities. Both baseline and BrainMoE are pretrained with subjects 2-7 and finetuned with subject 1 on the entire 40 sessions. The final train and test losses, cosine similarity, and Mean Squared Error (MSE) during finetuning are listed in Table E. Given the evidence that the performance of BrainMoE is better than the single expert MindEye2, there is potential for BrainMoE to expand to visual decoding.\n\n---\n*Table E*\n\n||MindEye2|BrainMoE|\n|-|-|-|\n|Train loss|9.639|**7.994**|\n|Test loss|11.142|**9.405**|\n|Cos. Sim.|0.778|**0.840**|\n|MSE|0.301|**0.261**|\n\n[1] 10.1038/s41593-021-00962-x\n\n[2] 10.48550/arXiv.2403.11207\n\n[3] 10.48550/arXiv.2210.06681\n\n[4] 10.48550/arXiv.2402.03226"}, {"review_id": "UtQeKjJnzY", "reviewer": "Reviewer_hbEn", "summary": "The manuscript proposes BrainMoE, a mixture-of-experts (MoE) framework for building brain foundation models by leveraging task-based as well as resting-state fMRI data across multiple cognitive states. Unlike prior models trained primarily on resting-state data, BrainMoE stratifies pretraining by cognitive states, training separate expert models for each of 12 distinct behavioral tasks. These expert embeddings are then fused via a cognition adapter and fine-tuned to support diverse downstream tasks (e.g., disease classification, behavior prediction, sex classification) across varied datasets and preprocessing pipelines. Empirical results across seven datasets demonstrate consistent and often substantial performance increase over BrainMass and BrainJEPA.", "questions": "1. The Methods suggest that the number of experts, N, equals the number of cognitive tasks. Why are there 36 experts in the end? There is a contradiction in the description.\n\n2. Training data are primarily from UKB, an aging population. How does that impact results on ABIDE,  a primarily young population.\n\n3. Do all three datasets have all 12 tasks?\n\n4. In Table 1, HCPA and HCPYA are also considered downstream tasks. I thought those are tasks used for pre-training the model. Is it still fair to evaluate on the 3-task or 7-task classification as a downstream task?", "limitations": "In my experience, study different (or scanner difference) is a major hurdle for generalizability. The authors should acknowledge this limitation and also the lack of handling other confounding effects in the downstream classification.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 4, "strengths_and_weaknesses": "Strengths:\n\n* Incorporating task-fMRI in building foundation models and using MoE to resolve the heterogeneous connectivity patterns linked cognitive states from different tasks is a well justified reasonable technical proposal\n\n* Experiments clearly indicate the performance gain from the MoE model.\n\nWeaknesses:\n\n* Writing of the Methods section can be improved. Although I understand the overall rationale, I'm still confused about Section 3.3 on how to select top-k experts. The confusion partly stems from the discrepancy between figure 3 and figure 4 that seemingly suggest different architectures of the model.\n\n* The cognitive classification task seems to be a trivial task. I'm not sure how it can help self-supervised representation learning. Even when you train 12 separate encoders on exactly the same training data, you can freely get 12 versions of embeddings (e.g., just by applying a translation to the embeddings, you have a naively constructed different embeddings). Now the data for different experts are different, and the experts are trained independently, so I'm not sure why is there a need to further enforce a classification loss, a trivial task.\n\n* Model interpretation is non-informative. Yes, some regions are picked, and with more experts, more regions are picked, but this is not something particularly interesting. I'd be interested to know whether each expert is linked to a specific cognitive construct as promised, and for each disease which expert (cognitive domain) is prioritized.", "was_revised": true, "final_justification": "The authors reasonably addressed my concerns. My original rating was already high. The novelty and results warrant publication.", "rebuttal": "## **W1**\n\nWe appreciate your suggestion, and we are committed to thoroughly revising the method section. Specifically, we will highlight the innovation of BrainMoE and connect Figs 3 and 4 with a smooth description as in this rebuttal: We illustrated the overall framework of BrainMoE in Fig. 3, including pre-training and fine-tuning, respectively. Fig. 4 illustrates the architecture of novel blocks shown in Fig. 3, particularly the brain expert and the cognition adapter. Figure 4 (a) shows the pre-training of brain experts corresponding to the individual expert in Figure 3 (a). On the other hand, Fig. 4 (b) shows the fine-tuning for downstream, and it corresponds to Fig. 3 (b), where the cognition adapter is the main block in the fine-tuning stage. We will combine Fig. 3 and Fig. 4 as one figure to clarify the method description.\n\n## **W2**\n\nThe motivation behind this is considering the variations of brain expert architecture to the extent of supervision type. The utilization of classification loss complements the gap between standard brain classification models, e.g., transformer-based [3][4], and the brain foundation models. Empirical evidence in our work shows that classification loss derived from standard brain modeling can benefit the performance of BrainMoE on clinical (Cog. Classif. in Table 3) and non-clinical applications (All-in-one in Table 4), indicating the potential of standard brain classification architectures for brain foundation modeling. We will clarify this motivation in the Introduction section and discuss the results in the Conclusion section.\n\n## **W3**\n\nHere, we collect the expert routing probabilities of BrainMoE on clinical datasets and count the combination of top experts (top 1 if the probability > 0.9, otherwise top 2). We can observe the most frequent combination (Table C for reviewer PTHj):\n - AD: [Working memory] 31.1%, [Rest, Motor] 20%\n - ASD: [Language, Relational (relationships between stimuli)] 14.9%, [Rest, Relational] 8.3%\n - PD: [Language, Relational] 20%, [Vismotor (color change and button press), Relational] 9.1%\n - SZ: [Gambling, Relational] 18.5%, [Facename (face and name recall)] 14.3%\n\nThe relatively more frequent activation of these expert combinations, e.g., [Language, Relational] for ASD and PD, suggests that BrainMoE captures complementary neural patterns that individual experts might miss.\n\nOn the other hand, combinations for different applications align with neuroscientific knowledge: \n - For AD, the dominance of [Working memory] aligns with that AD is commonly attributed to central executive impairment and frontal lobe dysfunction [1].\n - For ASD, [Language, Relational] mirrors impairments in relational memory processing, where individuals with ASD show difficulties in forming relations between items and encoding relational but not item information [2], along with the language network showing atypical functional connectivity [3].\n - For PD, [Vismotor, Relational] reflects the visuomotor processing deficits, including impaired decision-making cascades and altered cortical sensorimotor processing [4].\n - For SZ, those combinations reflect deficits in reward learning and decision-making processes, as well as impaired facial processing and recognition abilities that are characteristic of the disorder [5].\n\n[1] 10.1016/j.cortex.2010.12.002\n\n[2] 10.1002/aur.1493\n\n[3] 10.1002/aur.2171\n\n[4] 10.3390/brainsci13081173\n\n[5] 10.3389/fneur.2019.00990\n\n## **Q1**\n\nSorry for the confusing description of the expert number. The number of experts (N) is derived from the number of distinct cognitive states of arbitrary input type (FC or BOLD) required by an arbitrary expert model architecture. Expert number is not strictly equal to the number of cognitive tasks but $N_{arch}\\times N_{cog}$, where $N_{arch}$ is the number of available model architecture and $N_{cog}$ is the number of distinct cognitive states. In practice, there are three types of expert architecture (BrainMass, BrainJEPA, and Cog. Classif.) implemented in our BrainMoE All-in-one, so that $3\\times 12=36$. We will revise the description in L.113-L.120 to clarify the expert number.\n\n## **Q2**\n\nHCPYA has an age range 22 to 35 including young adults. But yes, given ABIDE has a younger population (age < 22), the performance of BrainMoE on ABIDE (ASD) is impacted as the worst among binary classifications (Table 1).\n\n## **Q3**\n\nNo. UKB has 2: Resting and task-Hariri. HCPA has 4: Resting, task-Carit, task-Facename, and task-Vismotor. HCPYA has 7: Motor, gambling, language, social, relational, emotion, and working memory. In total, there are 12 different cognitive states.\n\n## **Q4**\n\nIt is a fair comparison on HCPA and HCPYA datasets since the same cohorts of data were used for different models in the pre-training stage. Baselines and BrainMoE both have the same 68k pre-training data with the same number of training iterations. Although there are multiple experts in BrainMoE, each expert only sees a specific cognitive state, and each sample only has one cognitive state. Thus, BrainMoE has no additional pre-training epochs or iterations than baseline models, resulting in a fair comparison.\n\n## **Lim.1**\n\nWe appreciate your suggestion, and we are committed to thoroughly revising the ‘Limitations’ section with the discussion of confounding effects from study/scanner difference."}, {"review_id": "J7yJapvtJE", "reviewer": "Reviewer_Yxzq", "summary": "The idea is to use separate experts for separate cognitive states/tasks. In addition to MSE the authors also use cognitive state classification. Traditional models use the same weights across wildly different fMRI tasks. The model demonstrates substantial performance gains over baselines across 7 datasets and 3 preprocessing pipelines.", "questions": "How do we know the performance boost is not due to increase in parameters?\nHow do you think this is better compared to task embeddings?\nWhy was a Transformer decoder chosen over other scalable alternatives like MLPs or lightweight convolutional heads?\nDid the authors observe any conflicts or redundancy between experts trained on similar cognitive states?\nCould BrainMoE generalize to modalities beyond fMRI (e.g., EEG) if modality-specific experts are trained?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Strength\n- This is a novel application of mixture-of-experts to fMRI data.\n- Shows considerable performance improvement in multiple datasets/tasks\n- Good evaluation of different pretraining objectives\n- Compatible with existing foundational models and input types\n- Good reproducibility\n\n\n###  Weaknesses\n- Evaluation should be done primarily cross-subject\n- Higher computational cost", "was_revised": true, "final_justification": "I maintain my original acceptance recommendation, as it is already high, and suggesting a strong accept would need groundbreaking results with much higher impact.", "rebuttal": "## **W1**\n\nThank you for the comment. Our evaluations are exactly cross-subject. Each fMRI in the five clinical datasets corresponds to one unique subject, and the evaluations are subject-wise (L.191-L.206). Since two non-clinical datasets are for cognitive state recognition, there are multiple cognitive states per subject, but the train/test split in our multi-fold cross-validation is still subject-level (L.207).\n\n## **W2**\n\nThe running time of BrainMoE has fulfilled the real-time requirement for fMRI applications, which takes a much longer time (10-15mins) than BrainMoE (133-287ms) for data acquisition using a common 3T scanner [1].\nAdditionally, BrainMoE has been implemented with the pre-computation of cognitive embeddings from complex experts to amortize the high computational costs from multiple brain experts. For instance, BrainJEPA experts have high FLOPs (5.32T) and compute cognitive embeddings prior to BrainMoE finetuning, while BrainMass experts with much lower FLOPs (26.96M) have the cognitive embeddings computed on-the-fly. This trick enhanced efficiency as listed in Table 5, where BrainJEPA with the trillion-level FLOPs costs less time than BrainMass with the million level FLOPs. We will include the discussion of this possible amortized efficiency pruning in the “Limitation” section of the final version.\n\n[1] Kumar, V. A., et al. \"Recommended resting-state fMRI acquisition and preprocessing steps for preoperative mapping of language and motor and visual areas in adult and pediatric patients with brain tumors and epilepsy.\" American Journal of Neuroradiology 45.2 (2024): 139-148.\n\n## **Q1**\n\nParameters are increased $N$ times compared to the previous single expert model, where $N$ is the number of experts. However, increasing the parameter amount by fusing multiple experts naively, e.g., a late fusion MoE [2], sometimes cannot boost the robustness of brain modeling (80.70 -> 73.33 for AD prediction). The table below shows the seven downstream applications using baseline (BrainMass), the late fusion MoE, and our BrainMoE with the parameter amount of each model. BrainMoE, in contrast, consistently has the best performance.\n\nNote that late fusion is a strategy for handling multimodal data where the final output is the weighted sum of predictions from each independent expert selected by the router. Unlike BrainMoE has the cognition adapter to implicitly utilize multiple expert embeddings via Transformer, the late fusion directly combining expert task embeddings suffers from a data unbalance issue for different domain experts (Table A for Reviewer ZhRL). Therefore, the performance boost of BrainMoE is not only advanced by more parameters but also by our new ML component, the cognition adapter.\n\n||ADNI (AD)|ABIDE (ASD)|PPMI (PD)|Taowu (PD)|SZ (SZ)|HCPA (3-task)|HCPYA (7-task)|\n|-|-|-|-|-|-|-|-|\n|Baseline (34M)|80.70$_{\\\\pm7.85}$|67.81$_{\\\\pm3.91}$|59.77$_{\\\\pm14.22}$|68.00$_{\\\\pm21.46}$|77.63$_{\\\\pm8.56}$|90.63$_{\\\\pm0.74}$|81.27$_{\\\\pm1.27}$|\n|Late fusion MoE (408M)|73.33$_{\\\\pm10.87}$|69.89$_{\\\\pm306}$|61.11$_{\\\\pm15.29}$|**91.24$_{\\\\pm11.72}$**|76.95$_{\\\\pm9.50}$|94.96$_{\\\\pm2.64}$|88.58$_{\\\\pm3.39}$|\n|BrainMoE (709M)|**81.23$_{\\\\pm11.00}$**|**70.26$_{\\\\pm3.40}$**|**62.97$_{\\\\pm13.94}$**|88.57$_{\\\\pm12.51}$|**83.10$_{\\\\pm11.33}$**|**96.67$_{\\\\pm0.77}$**|**93.19$_{\\\\pm0.72}$**|\n\n[2] Han, Xing, et al. \"Fusemoe: Mixture-of-experts transformers for fleximodal fusion.\" Advances in Neural Information Processing Systems 37 (2024): 67850-67900.\n\n## **Q2**\n\nBrainMoE approach uses cognitive embeddings so that the cognition adapter can learn from the latent space, which is the information bottleneck between data space and task space (Fig. 1). The latent space is a lower-dimensional representation of the input data, is designed to capture the most essential and relevant information from the original data while discarding less important details [3]. The task embeddings, on the other hand, capture the characteristics of a specific downstream application instead of various cognition-related features in the latent space by domain experts, while either clinical or non-clinical brain modeling is highly relevant to cognitive function or dysfunction. The less robust performance of late fusion MoE further provides empirical evidence to support that cognitive embeddings are better than task embeddings.\n\n[3] Wang, Shiye, et al. \"Self-supervised information bottleneck for deep multi-view subspace clustering.\" IEEE Transactions on Image Processing 32 (2023): 1555-1567.\n\n## **Q3**\n\nThe scalability of MLP has been analyzed in Appendix Figure 8. Curves of the train loss at the 200th epoch by MLP are increasing with more parameters, using more layers and hidden channels in 3 datasets, demonstrating the bad scalability in brain modeling. In contrast, our Transformer-based adapter maintains a decreasing train loss at the 200th epoch when there are more parameters. Note that lightweight convolutional heads are specific for image data, while FC is structured data, and BOLD is a timeseries data, so that convolutional heads cannot be adapted to our cases.\n\n## **Q4**\n\nNo, there are no observable identical or conflicting cognitive embeddings from different brain experts in the observation of the correlation between cognitive embeddings. The table below is the mean value of Pearson’s correlation between BrainMass experts specified for 12 different cognitive states, where the maximum correlation is 0.51 [Language, Working memory] and the minimum is -0.47 [Vismotor, Rest]. The absolute value of correlation is mostly less than 0.5, indicating experts are not in conflict or redundant to each other.\n\n|Expert pair|Mean corr|Expert pair|Mean corr|Expert pair|Mean corr|Expert pair|Mean corr|\n|-|-|-|-|-|-|-|-|\n|vismotor-carit|0.02|carit-facename|\\-0.22|rest-gambling|\\-0.14|wm-social|\\-0.1|\n|vismotor-language|0|carit-motor|0.28|rest-facename|0.22|wm-hariri|0.22|\n|vismotor-rest|**\\-0.47**|carit-social|\\-0.3|rest-motor|\\-0.01|wm-relational|\\-0.11|\n|vismotor-emotion|\\-0.23|carit-hariri|0.02|rest-social|\\-0.09|gambling-facename|\\-0.12|\n|vismotor-wm|\\-0.31|carit-relational|0.2|rest-hariri|\\-0.1|gambling-motor|0.22|\n|vismotor-gambling|\\-0.04|language-rest|\\-0.28|rest-relational|\\-0.23|gambling-social|0.16|\n|vismotor-facename|0.13|language-emotion|\\-0.03|emotion-wm|0.39|gambling-hariri|\\-0.16|\n|vismotor-motor|\\-0.12|language-wm|**0.51**|emotion-gambling|\\-0.27|gambling-relational|\\-0.16|\n|vismotor-social|0.38|language-gambling|0.01|emotion-facename|\\-0.19|facename-motor|0.44|\n|vismotor-hariri|0|language-facename|\\-0.14|emotion-motor|\\-0.35|facename-social|\\-0.19|\n|vismotor-relational|0.21|language-motor|0.06|emotion-social|\\-0.17|facename-hariri|\\-0.17|\n|carit-language|0.04|language-social|\\-0.04|emotion-hariri|0.3|facename-relational|\\-0.02|\n|carit-rest|\\-0.47|language-hariri|\\-0.05|emotion-relational|0.28|motor-social|\\-0.24|\n|carit-emotion|\\-0.14|language-relational|\\-0.24|wm-gambling|\\-0.19|motor-hariri|\\-0.27|\n|carit-wm|\\-0.23|rest-emotion|0.34|wm-facename|\\-0.15|motor-relational|\\-0.16|\n|carit-gambling|0.12|rest-wm|\\-0.02|wm-motor|\\-0.2|social-hariri|\\-0.02|\n|social-relational|\\-0.13|||||||\n|hariri-relational|0.34|||||||\n\n## **Q5**\n\nYes, it could. We selected a multimodal (fMRI+EEG) visual stimuli dataset [4] for the evaluation. There are 388 fMRI-EEG pairs from 22 healthy subjects under 8 cognitive states (age$\\in$[23,51], F:M=1:1). The EEG expert is implemented with the pre-trained CBraMod [5]. Since the preprocessed fMRI in this dataset was using the Shaefer atlas and only Cog. Classif. experts in our work have been pretrained with Shaefer400, BrainMoE was evaluated on this FMRIEEG dataset with one CBraMod expert and 12 Cog. Classif. experts. This experiment is a subject-level 5-fold cross-validation. Results of cognitive state classification, sex classification, and age regression are shown in the tables below. Better performance by multimodal BrainMoE further indicates the value of our approach.\n\nThe table below shows the cognitive state classification performance of BrainMoE, where 'Prec' is precision, and 'Rec' is recall.\n\n---\n*cognitive state*\n\n|Expert|Accuracy|F1|Prec|Rec|AUC|\n|-|-|-|-|-|-|\n|fMRI|67.72$_{\\pm5.50}$|67.66$_{\\pm5.74}$|73.14$_{\\pm6.44}$|67.72$_{\\pm5.50}$|**90.45$_{\\pm6.54}$**|\n|EEG|69.20$_{\\pm2.19}$|68.71$_{\\pm1.46}$|71.35$_{\\pm1.20}$|**69.24$_{\\pm2.19}$**|84.22$_{\\pm1.16}$|\n|fMRI + EEG|**69.21$_{\\pm4.06}$**|**68.73$_{\\pm3.72}$**|**73.41$_{\\pm1.99}$**|69.21$_{\\pm4.06}$|87.09$_{\\pm5.51}$|\n\nThe table below shows the sex prediction performance of BrainMoE.\n\n---\n*sex*\n\n|Expert|Accuracy|F1|Prec|Rec|AUC|\n|-|-|-|-|-|-|\n|fMRI|63.43$_{\\pm4.60}$|63.67$_{\\pm5.16}$|65.61$_{\\pm5.84}$|63.43$_{\\pm4.60}$|63.40$_{\\pm7.39}$|\n|EEG|65.53$_{\\pm3.19}$|65.39$_{\\pm2.33}$|67.91$_{\\pm3.32}$|65.53$_{\\pm3.19}$|**65.68$_{\\pm7.71}$**|\n|fMRI + EEG|**66.47$_{\\pm3.54}$**|**65.47$_{\\pm5.38}$**|**70.29$_{\\pm4.70}$**|**66.47$_{\\pm3.54}$**|65.00$_{\\pm6.10}$|\n\nThe table below shows the age regression performance of BrainMoE, where 'Abs. Corr.' is the absolute Pearson's correlation between ground truths and predictions, and 'MSE' is the mean squared error.\n\n---\n*age*\n\n|Expert|MSE|Abs. Corr.|\n|-|-|-|\n|fMRI|8.05$_{\\pm5.58}$|0.512$_{\\pm0.204}$|\n|EEG|8.26$_{\\pm5.97}$|0.475$_{\\pm0.179}$|\n|fMRI + EEG|**7.99$_{\\pm5.53}$**|**0.523$_{\\pm0.152}$**|\n\n[4] Telesford, Qawi K., et al. \"An open-access dataset of naturalistic viewing using simultaneous EEG-fMRI.\" Scientific Data 10.1 (2023): 554.\n\n[5] J. Wang et al., ‘CBraMod: A Criss-Cross Brain Foundation Model for EEG Decoding’, in The Thirteenth International Conference on Learning Representations, 2025.\n\n## **Paper format 1**\n\nWe appreciate your suggestion, and we are committed to thoroughly revising the manuscript. Specifically, we will highlight the innovation of BrainMoE and provide more comprehensive supporting results as shown in this rebuttal.\n\n\n## **Paper format 2**\n\nWe will revise the font color to dark green."}]

Author rebuttal:

Meta-review: The manuscript introduces BrainMoE, a mixture-of-experts (MoE) framework for developing brain foundation models. BrainMoE leverages task-based and resting-state functional magnetic resonance imaging (fMRI) data across multiple cognitive states. Unlike prior models that were primarily trained on resting-state data, BrainMoE stratifies pretraining by cognitive state. It trains separate expert models for each distinct behavioral task. These expert embeddings are then fused via a cognition adapter and fine-tuned to support various downstream tasks, such as disease classification, behavior prediction, and sex classification, across several datasets and preprocessing pipelines. Empirical results across seven datasets demonstrate a consistent and often substantial increase in performance over current methods such as BrainMass and BrainJEPA.
Overall, the reviewers were pleased with the original idea and the extensive empirical validation. However, they also raised concerns about missing baselines, the lack of interpretability of the method, and inference cost.
Many of these concerns have been successfully addressed by the authors. Ultimately, the paper is sound, and the empirical results are interesting. However, while the method is robust and well-executed, it is not very novel, as it largely adapts existing Mixture-of-Experts concepts from other domains without groundbreaking innovations in the neuroimaging context. The paper does not provide significant neuroscientific insights.
Nevertheless, there is a consensus for acceptance.
Decision: Accept (poster)