Brain-Semantoks: Learning Semantic Tokens of Brain Dynamics with a Self-Distilled Foundation Model
Authors: Sam Gijsen
Marc-Andre Schulz
Kerstin Ritter
Venue: ICLR 2026 Poster (2026)
Keywords: neuroscience
neuroimaging
SSL
self-supervised learning
representation learning
foundation model
Primary area: applications to neuroscience & cognitive science

Abstract: The development of foundation models for functional magnetic resonance imaging (fMRI) time series holds significant promise for predicting phenotypes related to disease and cognition. Current models, however, are often trained using a mask-and-reconstruct objective on small brain regions. This focus on low-level information leads to representations that are sensitive to noise and temporal fluctuations, necessitating extensive fine-tuning for downstream tasks. We introduce Brain-Semantoks, a self-supervised framework designed specifically to learn abstract representations of brain dynamics. Its architecture is built on two core innovations: a semantic tokenizer that aggregates noisy regional signals into robust tokens representing functional networks, and a self-distillation objective that enforces representational stability across time. We show that this objective is stabilized through a novel training curriculum, ensuring the model robustly learns meaningful features from low signal-to-noise time series. We demonstrate that learned representations enable strong performance on a variety of downstream tasks even when only using a linear probe. Furthermore, we provide comprehensive scaling analyses indicating more unlabeled data reliably results in out-of-distribution performance gains without domain adaptation.

Published as a conference paper at ICLR 2026
BRAIN-SEMANTOKS:
LEARNING SEMANTIC TOKENS OF BRAIN DYNAMICS WITH A SELF-DISTILLED FOUNDATION MODEL
Sam Gijsen1.2.3+ Mare-Andre Schulz Kerstin Ritter1.23
Hertic Institute for Al in Brain Health, University of Tübingen, Germany
TTübingen AT Center, University Tübingen, Tübingen, Germany
Charité Universitätsmedizin Berlin, Department of Psychiatry and Psychotherapy, Berlin, Germany
sam. g1 jswnnchaaittt...m marc-andre, www.lsscharitoo...o kerttti..tttttrrtn--taaaaaaaeee
ABSTRACT
The development of foundation models for functional magnetic resonance imag- ing (IMRI) time series holds significant promise for predicting phenotypes related to discase and cognition. Current models, however, are often trained using mask- and-reconstruct objective on small brain regions. This focus on low-level informa- tion leads to representations that are sensitive to noise and temporal fluctuations, necessitating extensive fine-tuning for downstream tasks. We introduce Brain- Semantoks, a self-supervised framework designed specifically representations of brain dynamics. Its architecture is built on two core innovations: a semantic tokenizer that aggregates noisy regional signals into robust tokens rep- resenting functional networks, and self-distillation objective that enforces repre- sentational stability across time. We show that this objective is stabilized through a novel training curriculum, ensuring the model robustly learns meaningful features from low signal-to-noise time series. We demonstrate that learned representations enable strong performance variety of downstream tasks even when only using a linear probe. Furthermore, we provide comprehensive scaling analyses indicat- ing more unlabeled data reliably results in distribuuit performance gains without domain adaptation.
learn abstract
INTRODUCTION
The investigation of brain dynamics has been a cornerstone of neuroscience, progressing our under- standing of human cognition, disease, and aging. Functional magnetic resonance imaging (fMRT) has been an instrumental modality in this endeavor; its surement relates to local changes in brain activity, and its non-invasive nature has made it primary tool across numerous research fields (Ogawa et al.] 1990; Logothetis) 2008). Despite being ex- tremely nigh-dimensional, fMRI data is often collected in limited samples, which can severely strain the potential insights (Button et al. 2013)] Poldracck et al. 2017). This challenge motivates a shift towards data-driven representation learning, where progress in self-supervised learning (SSL) can enable the training of highly capable 'foundation' models from large quantities of unlabeled data, promising a new way forward for neuroimaging analysis.
(BOL.) mea-
Current [MRI foundation models, however, adapt eevonstruction-centte paradigms from NLP and vision, such as masked signal prediction (Caro et al. 2023 Wang et al. (2025). While latent-space reconstruction (e.g., JEPA; Dong et al. (2024)) avoids modeling the substantial noise in the BOLD signal, these models still focus on lewelevel, regional information. We argue this objective mis- aligned with predicting stable, high-level phenotypes, as the resulting representations require ex- tensive fine-tuning for such tasks, reducing the utility of a foundation model. This dependency is particularly problematic for fMRI, where transfer is challenged by significant variations across datasets in participant cohorts, hardware, and acquisition protocols.
We make code and models available /ttpp::///tttie wwm. jsen /Brai Semantoks

---

Published as a conference paper at ICLR 2026
To address these issues, we hypothesize that effectively predicting stable phenotypes requires shift from reconstruction to abstraction. The goal should not be to perfectly encode the BOLD signal, but to abstract away from it to find the underlying phenotypic signature. We propose Brain- Semantoks, a foundation model built on a strong neuroscientific inductive bias to learn such abstract representations. Our approach starts the input level, recognizing that self-attention mechanisms, central to modern transformers, perform best on sequences of low-noise, semantic tokens, akin to words in natural language. Time series of individual, small regions are poor tokens in this regard as they are noisy and lack high-level meaning. We therefore introduce a semantic tokenizer, which aggregates information from regions within a functional brain network (e.g., default mode network) into a single, robust token. This creates a shorter, more computationally efficient, and semantically meaningful sequence for the transformer to operate on.
With these semantic tokens, we then shift the learning objective itself. Instead of focusing on the reconstruction of masked signals, -rain-Semmntoks is trained using a self-distillation objective to produce stable, summary representation across different temporal views of the same scan (Grill ct al, 2020; Caron et al.] 202I). This explicitly trains the model to capture a stable, high-level representation of individual's brain dynamics, which we expect to transfer better across data distributions. However, while conceptually highly suitable, we found that applying this objective to low signal-to-noise fMRI data can lead to training instability, where the model converges on poor, simple solution. To solve this, we introduce a Teacher-guided Temporal Regularizer (TTR), a novel training curriculum active only at the start of training. This regularizer guides the model to first learn the time-averaged signature of each network before modeling more complex temporal variations, ensuring robust and meaningful pretraining convergence. The resulting representations are particularly powerful for linear probing, indicating they are well-disentangled and broadly useful without task specific fine-tuning.
Contributions. Our contributions are threefold. First, we propose new pre-training approach that prioritizes abstract representations over signal reconstruction, enabled by a novel semantic tokenizer and maacher-guided Temporal Regularizer (TTR) to stabilize training. Second, we introduce Brain- Semantoks, a foundation model trained with this method that achieves state-of-the-ar performance on diverse downstream tasks under rigorous linear probing protocol. Finally, we provide the first detailed scaling analysis for fMRI foundation models, showing consistent duiti--dis performance gains without domain adaptation.
2 RELATED WORK
Self-Supervised Learning for MRI. Much early self-supervised learning work focused on recon- struction using auto-encoders (Han et al. 2019 Pinaya et al.) 2019 Kim et 2021), using relatively limited data. The first effort to build an IMRI foundation model similarly adapted reconstruction- based objectives popular in other domains. BrainLM (Caro et al., 2023) employs a masked mod- cling objective to reconstruct the BOLD signal in input space. While effective, this approach risks modeling the substantial noise inherent in fMRI data. More recent work like Brain-JEPA (Dong let al. 2024) mitigates this by predicting masked representations in a latent space, thereby learning to ignore noise. Meanwhile, NeuroSTORM operates on 4D voxel data, performing spatio-temporal reconstruction (Wang et al.) (2025). However, all these methods remain fundamentally focused on predicting low-level information. In contrast, models like BrainMass learn from static functional connectivity matrices (Yang et al., 2024), ignoring the rich temporal dynamics central to our work.
Self-Distillation Learning. Self-distillation has proven highly effective for learning semantic fea- tures, improving upon contrastive learning methods (Chen et al. (2020). Seminal works include MoCo (He et af. 2020), BYOL (Grilll et al. 2020), and DINO (Caron et al., 2021 Oquab et al., 2023 (Siméoni et a 2025) demonstrated that student network can learn powerful, linearly sepa- rable representations by matching the output of a teacher network (a momentum-updatte version of itself) across different views of a sample. This approach avoids "representational collapse" without requiring negative samples or a reconstruction loss. The iBOT framework (Zhou et al., 2021) further advanced this by integrating masked-tokee prediction objective within the distillation framework, enabling the model to learn both a global summary representation as well as rich, context-awe local features. Recent work by Wu et al. (2025) makes important progress in understanding what
2

---

Published as a conference paper at ICLR 2026
is necessary to prevent collapse and significantly simplifying the approach, reducing the number of hyperparameters.
Regions
Stabus
Pheteering
G: Semartic
Robi
Student
sewwe.....
Teacher
Goal
Aunggn rehwork reppang
Fisilely
Cyck
BY
Pluli POO
infrence
Brain Networka
Outact Nethorky
Crnk
Figure 1: Brain-Semantoks. A student-teacher architecture is used to learn stable brain dynamics representations across time by aligning long temporal views. The semantic tokenizer (G) is used to produce robust tokens of functional brain networks, which serve input to transformer encoder (f). Three losses are used during pretraining: a temporary regularisation loss for stability (CttR), within view, latent space prediction loss of masked tokens (Crok), and a global cross-view loss to learn high-level, semantic representation (LCLs).
METHOD: BRAIN-SEMANTOKS
Our proposed framework, Srain-Semantoks, learns abstract and temporally stable representations from IMRI time series. The methodology is built upon three core innovations designed to address the unique challenges of fMRI data. First, we introduce a paradigm performing self-distillation across time, that explicitly trains for high-level representations suitable for transfer learning. Second, we develop a semantic tokenizer with a strong neuroscientific inductive bias create a robust and meaningful input space for our encoder model. Finally, we introduce training curriculum that stabilizes the learning objective, ensuring convergence on low signal-to-noise data. The framework uses student-teacher architecture, depicted in Figure
3.1 A SELF-DISTILLATIO FRAMEWORK FOR SEMANTIC REPRESENTATIONS
The primary goal of a foundation model is to learn representations that are broadly applicable with- out requiring tpsk-ipe fine-tuning To achieve this with fMRI, model must learn capture the stable, underlying phenotypic signature of a subject, abstracting away from transient noise and acquisition-sseeifie artifacts.
Input Data and Augmentation: We represent a subject's 's fMRI time series as matrix X E RCXT where C is the number of brain regions of interest (ROIs) and T is the number of time points. To generate different views of the same underlying brain dynamics we create two long temporal segments of length Terop < T resulting in two views, (1) and (2). Unlike computer vision, a large set of intuitive augmentations distillation across time and only lightly further augment the views with corrupting transformations: randomly select a fraction of channels (r.) and contiguous timepoints (7t) and set them to zero, add gaussian noise sampled with μ = 0 and a = To, and finally scale the time series amplitude XT,-
not available for fMRI. We therefore mainly rely self-
student-teacher architecture enforce representational consistency across these two views. The student network (0.). is trained to match the output of the teacher network ((t). The teacher provides a stable regression target as its weights are an exponential moving average (EMA) of the student's weights:
Student-Teacher Framework:
We
0, +- a0, + (1,
(1)
where the momentum coefficient a gradually increases during training. This forces the student to learn high-level representations which are stable across time.
3

---

Published as a conference paper at ICLR 2026
3.2 SEMANTIC TOKENIZER
We posit that standard tokenization such as a direct linear projection of ROI signals is suboptimal for fMRI data. This approach creates overly long sequences of noisy, low-level tokens that hinder a transformer's ability to learn meaningful long-range dependencies. Our innovation a semantic tokenizer, G(Φ). that addresses this by creating compact and robust input space grounded in core neuroscientific prior: the brain's 's organization into functional networks. For a given fMRI sean ( our tokenizer uses N independent, network-specific modules, In. each operating on the time series of a single network. Specifically, each In processes the subset of ROIs belonging to network n, denoted X, E RC.xT, to produce sequence of semantically rich, D-dimensional tokens, (()))
To model the BOLD signal's complex temporal structure, each module gn first divides its input time series into P relatively long temporal patches Xp (TTPP) inspired by the temporal stability of macro-scale brain dynamics (Aflen et al., 2014] Vidaurre et al. 2017). Within each patch, a multi- scale convolutional filter bank, composed of standard convolutional branch, Can and structured convolutional branch, (.)) (Li et al. (022). captures hierarchical temporal patterns.
For more details on the implementation these branches, please see Appendix A..
A final D-dimensional token embedding for each patch is generated via the transformation: token LayerNorm (AvgPool (GELU (Z))); (xp)))((:)))))).. The complete token tensor is then formed by concatenating the outputs from all network modules, Z [ 2 result- ing in final tensor of shape R xxxx, This approach yields compact and semantically rich input that provides the transformer encoder with better starting point for learning, which is crucial for the stability of the self-distillation framework.
3.3 TRANSFORMER ENCODER AND MASKING
The sequence of hetwork-token Z is flattened into sequence of length N P. We add sinusoidal positional embeddings to encode the temporal order of the patches and a learnable network-specific embedding for each of the N networks encode their identity. Finally, learnable [CLS] token is prepended to the sequence, yielding sequence of length N x 1.
The student and teacher models each consist of two core components: transformer encoder back- bone, f(0), and a projection head, h(v). The backbone, f, processes the input token sequence to produce network and [CLS] embeddings. These embeddings are then processed by the projector, h, into the space where the distillation loss is minimized. The projector is used only during pre- training and is discarded thereafter, while the transferable representations are the outputs from the teacher's s encoder backbone, (O.).
The student network, fs (O.),,, receives a masked version of the token sequence. Masking is deter- mined by a binary mask ( E {0, 10,1)NNx NxP, which replaces tokens with learnable mask embed- ding. The teacher network always receives the full, unmasked sequence. To reduce the degree to which the model can rely on simple, interpolative relationships to predict masked tokens, we per- form 'slice masking'. Specifically, (networks by temporal patches) and mask out entire 'slices' rather than random individual tokens. We randomly select one of two strategies work slicing, masks entire rows, hiding all temporal data for one or more selected networks. Second, temporal slicing masks contiguous block of entire columns, hiding information from all networks for a specific period. By masking large, contiguous parts of data, we force the model to learn more complex relationships between networks and across time.
treat the input
as a 2D matrix of size N x P
sample-level to increase data diversity. The first, net-
3.4 PRETRAINING ORJECTIVE
Finally, we propose a multi-comppnent objective function that includes a curriculum to ensure stable training. All loss components are computed using the outputs of the projection heads (h, and hı), which are denoted as 2, and 2, Following recent work simplifying DINO (Wu et al, 2025). we regularize with coding rate term to prevent representation collapse.
4

---

Published as a conference paper at ICLR 2026
To learn high-level, stable representation of the brain time series, enforce consistency between the [CLS] tokens across two views. The loss is bidirectional and regularized:
(2) LCLS = d(> ,,css + d( c,,)ss -
(2)
where d the squared Euclidean distance and E is the expectation over data. Adopting the SIMDINO
implementation (Wu et al., 22225), we use the coding rate regularizer R. (2)) = 1/ log det(1 + 2 2)
with 0.05 and where prey is the projected feature dimension. This prevents subspace collapse by maximizing the determinant of the batch-estimated covariance E. To normalize regularization
strength across varying batch sizes B, we adopt the balancing heuristic
p
D
Network Token Loss To promote the model learning rich, temporally-sensiiee representations, apply an auxiliary distillation loss on the network tokens that were masked in the student's input, guided by the 2D mask matrix B(r) (Zhou et al. 2021). This loss computed within each view:
N
TTk = (1
Σ
B(s) up
d(z s.mpp (.pp))
(3)
p=1
Due to the semantic tokenizer, this task is performed on a more semantic network-level, rather than on noisier, lower region-level.
Teacher-guided Temporal Regularizer
Although our tokenization strategy significantly improves pretraining effectiveness, we found that a direct application of these distillation objectives can still lead to poor solutions with noisy fMRI time series. We therefore develop a principled stabilizing curriculum based on this observation.
Specifically, as we observe that more compact token sequence aided convergence and yielded representations with better predictive performance, we guide the student network to first learn the time- representation of each network. Conceptually, this constrains the token space N x P towards + 1, which helps find a good initial representation which can thereafter be refined with temporal variability. This can be directly adopted in the distillation framework by using the teacher model to provide the network-speciffi targets. The summary token for each network n is computed by averaging its P patch embeddings from the transformer output:
in
p
Z
I,n.
(4)
p=1
The regularisation loss then applied across views to these summary tokens:
N (d(z!), (2) + d(z(2) (.)
N
CTTR = E
Σ
R& (Cov(Zann)
(5)
The contribution of this regularizer decayed zero early in training stabilize initial training without overly constraining the final learned solution.
Total Loss Function
The final training objective is a weighted sum of the three hierarchical loss components:
LTotal = LOLS + ATok + ATTRCTTR
(6)
where Aook and ÀTTR are scalar hyperparameters balancing the contributions of different levels of self-supervision. Following training, the student weights are discarded while the teacher weights are used for downstream evaluation.
4 EXPERIMENTAL SETUP
4.1 DATASETS
Pretraining Data
5

---

Published as a conference paper at ICLR 2026
We leveraged the largest 3T resting- fMRI corpus available for unlabeled pretraining. Specifi- cally, we use 39139 preprocessed recordings from the UKBioBank as well as the participant age and sex variables (UKB: Miller et al. (2016); application number 25163. We held out 1625 recordings for downstream evaluation.
We extract parcel-wise time series using the cortical Schaefer-400 atlas (Schaefer et al., 2018 sub- cortical Tian-III atlas (Tian et al.] 2020), and cerebellar Buckner-7 atlas (Buckner et al. 2011), yielding 457 total ROIs. We follow IMRI preprocessing conventions and apply a 0.01-0. HHz band- pass filter. Data normalization is a crucial step to aid transfer learning. Whereas prior work has relied on robust scaling (Caro al. 2023 Dong et al., 2024). which preserves ROI-specific 'DC' offsets resulting from the fMRI scanner. it impairs transferability to datasets which have less or no such offsets. We therefore applied z-scoring to each ROI per scan, As the UKB's temporal reso- lution (0.735s) is higher than most available datasets, we temporally downsample to 2s, resulting in 180 timepoints for the 6 minute recordings. Both normalization and resampling happen on the parcellated time series, meaning these are light operations that can be performed online during data loading and thereby ease transferability.
Downstream Tasks
We construct varied set of prediction tasks with differing sample sizes across multiple datasets (Appendix Table Continuous targets are binned into multi-class labels to enable direct compari- son between linear probing and finetuning.
For internal evaluation, we use UKB data for sex prediction and five-class age prediction (bins with equal sample sizes). For external evaluation: SRPBS (Tanaka et al. 2021 enables binary classi- fication of schizophrenia and major depressive disorder versus controls; ABIDE (Craddock et al.
2013
Di Martino et al.] (2044) provides autism-spectrum disorder classification; HBN (Alexander al. 2017) allows prediction of language (CELF) and cognitive (WISC) scores using three bins, plus out-of-distrbuttion demographic prediction; LEMON (Babayan et al. 2019) provides mood (MDBF) and cognitive task scores (CVLT, TMT) with three bins each. For scaling analyses, we additionally use demographic data from SRPBS and ADHD200 (Bellec et al. 2017). See Appendix A.2 for more detailed scale descriptions.
In total, we evaluate on four demographic, three clinical diagnosis, three cognitive/languaee, and one mood prediction task. Critically, these datasets vary participant cohorts, acquisition protocols, and preprocessing. Time series standardization consists of resampling to 2s, band-pas pass filtering, and 7-scoring.
4.2 IMPLEMENTATION
Training and Model Architecture
We sample temporal views of length Terop 100 timepoints. At sample level. apply light augmentations by randomly zeroing out a fraction of channels To ~ 2d[0, 0.1] and contiguous time- points T; (0 0.3), adding Gaussian noise with Ty ~ 2/0.8, 1.2] The semantic tokenizer G maps an input time series X E RRCTTerap to a token tensor PxDs, with patch length of 20. We define N functional networks, based on the Yeo 7-ntwork parcellation for the cortex (Yeo et al. 2011), with two additional networks com- prising all subcortical and cerebellar ROIs, respectively. Within each network-specific tokenizer gn. the standard and structured convolutional branches each output features of dimension Dj /2, which are then concatenated.
To 0.1, and scaling the amplitude by
The transformer encoder f uses dimensionality D, = 768 with 8 layers. The projection head an MLP with hidden layers (D,, = 1024) and an output layer projecting down to D = 128 dimensions and applying fz-normalization. The head is shared across all three distillation objectives. We find it suffices to set ATTR = 0.5 (i.e., weighting of LTTR) and cosine-decay this weighting to zero over the first 5% of training steps. We use slice-masking with Ao 0.5 Serighting-facton and a high masking ratio sampled from 2/ (0.65, 0.85). For all analyses we use the model checkpoint following 100 epochs of pretraining. We provide further optimization hyperparameters in Appendix Table 17 We jointly train the tokenizer and encoder end-to-end. The reduced memory usage of the tokenizer enables pretraining in under two hours on single GPU with less than 20 GB of memory.

---

Published as a conference paper at ICLR 2026
Table 1: Balanced accuracy (%) for IMRI time series foundation models using linear probes, Mean + standard deviations. Best results in bold, second-best underlined. 1Significantly worse than best model (p < 0.05, Bonf--oonerroni, Appendix Table T8p.
Model
ABIDE
IIN CELF
IBBN WISC
1IBN
HIBN Sex
UKB
UKB
SKPPS MDD
SRPPS SZ
BranLM
L.72
Brin JEPA
$2.92
41.50
3
39.81
2,08
63.90
1,45
30,00 2.12
K3.23
1.28
52.72
4.08
57.33
3.3.75
Bran Semantoks
68.13 2.14
42.18 2800
40.7 2.43
39.06
1.8.1
69.53 1 0.95
M.18 1 1.5
87.52 a 0.52
62.00
1.799
69.26
Table 2: Comparisons with supervised and finetuned baselines (Balanced Accuracy (%)). Best results in bold, second-best underlined. Significantly worse than best model (p < 0.05, Holm- Bonferroni, Appendix Table
Model
LKB-Age
UKB-Sox
HBN-Age
HBN-Sex
HBN-CBLF
HEN-W WISC
27.04 1 1.51
80.631
0.89
41.81
1.36
66.51 1 1.42
42.41 4: 2.91
39.79 1 2.91
BNT
20.481 # 1.08
77.91
1 3.37
22.59'
4 4.31
61.74"
9.69
42.40
4.99
38.53
2.94
26.851
37.671
1.41
65.22
42.45
ReainMoss
23.511
+ 1.69
69.72
+ 1.89
31.60"
+ 1.01
56.97
+ 1.08
36.93
+ 2.06
38.00
+ 2.83
BrainLM
30.261
1.66
85.751
1::1::::777 0.77
39.311
2.02
64.37
2.32
39.27
4.46
35.34
3.61
Brain JEPA
30.601
1.60
86.701
1.20
41.91
2.00
65.57
2.28
39.60
4: 3.50
35. 20
3.10
Brain-Senamoks
31.15
0.52
19.166
0.81
2.80
40.87
+ Finetune
33.91
0.97
#7.13 4: 0.57
39.41
3.77
09:31
0.71
42.59
1.33
40.82 + 1.51
Model
LEMON CVLT
LEMON MDBF
LEMON-TMT
ABIDB
SRPBS MDD
SRPBS SZ
Average
39.49 8.32
41.14
00.12 2.98
60.30
4.65
71.59
50.687
BNT
36.761 + 4.90
37.90 + 5.12
33.86
+ 6.13
58.38
6.51
57.60
+ 4.57
66.69 599 = 5.09
46.23'
Ball
39.54
+ 4:71
37 98
+ 5.82
40.30
7.52
64.89
4.08
59.50
+ 4.52
67. 122122--1
+ 4. 0.23
50.11
BrainMass
32.24
3.10
31.96
6.21
39.14
+
+
59.82
48.43
Brainl M.
37.81
+ 6.91
34.00 + 1.84
35.33
+ 3.78
53.911
1 2.23
54.29
+ 2.38
60.10"
: 5.79
47.48"
Brain-JEPA
6.58
35.48
9.58
4: 4.00
+:
SaminnSumamnoks
42.10
40.23 5.74
2.14
62.60 4.79
44.36 3.36
38.58 = 4.65
39.21
1.91
68.44 = 1.16
63,60
2.56
21.05 =: 4.30
Evaluation
We evaluate representations two settings: linear probing and full fineituning. For linear probing, we freeze the pretrained teacher encoder and train single linear layer top of its outputs assess raw representation quality. For fine-tuning, the entire model is updated to provide comparison with supervised baselines. All evaluations use ten cross-validatie repetitions while stratifying the target. At test time, we average predictions of 8 equally-spaced temporal crops for all methods. For Srain-Semantoks evaluations and ablations, pretrain using three random seeds and average their scores for each CV iteration to improve reliability.
The linear probing setup is standardized across models (Appendix Table 17 For SemannSSmmanoos, the input to the linear layer is the concatenation of the teacher's [CLS] token and the average of all network-patch tokens. For the linear probing comparisons. have access to filtered, preprocessed data. The ROI-specific offsets that BrainLM and Brain-JEPA were pretrained on are therefore not present, and we observe chance-level performance of these models on this dataset. We note that the normalization strategy used for Train-Semantok is more universally applicable however.
omit the LEMON dataset as we only
The two main baselines we compare against are BrainLM (Caro et al. (023)) and Brain-JEPA (Dong al, 2024), which both fMRI time series foundation models using parcellated data. Importantly, both models are pretrained solely the UKB, which is necessary for informative comparisons. Please see Appendix A. for further baseline model descriptions.
RESULTS
We evaluated Train-Semantoke on diverse set of downstream tasks, assessing its performance via linear probing against mate-of-the-an art foundation models and fully supervised methods. We then conducted extensive scaling and ablation studies to validate our architectural and training choices, and finally leveraged model's unique properties for built-in interpretability.
7

---

Published as a conference paper at ICLR 2026
Nep AOMONIAAAA
105 Nowerdly 1000
2097 10
Figure 2:
Scaling performance of linear probing following pretraining on increasing sample sizes. We compare within and oif-iistiibutionn scaling. RW: Random weights. ROI: We compare to using a linear layer for projecting single-ROI timeseries instead our semantic tokenizer.
Table 3:
fMRI prediction using balanced accuracy (%) on the Hariri emotion task.
Model
Linear Probing
Finetuning
Block (Pad)
Blocks (Cont.)
2 Blocks (Cut.)
Block (Pad)
Blocks (Cont.)
2 Blocks (Cat.)
Brain-JEPA
82.29 0.28
1.86
Brain-Semmantok (Ours)
93.84 + 0.36
94.34 0.75
96.50 = 0.15
96.89 + 0.73
97.85 0.86
97.70 = 0.80
5.1 DOWNSTREAM PERFORMANCE
A primary goal for foundation model is to produce representations that are directly useful for downstream tasks without extensive fine-tuning. We therefore prioritize evaluation using a rigorous linear probing protocol, where the pretrained model weights frozen.
As shown in Table foundation models, which are based on reconstruction objectives. Our model achieves the highest accuracy on eight of the nine tasks, often by a large margin. The improvements are particularly striking on challenging out-of-distribution clinical datasets, where Brain-Semantoks achieves large performance gains for predicting ASD, Schizophrenia, and MDD.
Prrin-Seemattks consistently and significantly outperforms existing fMRI
We compared the linear probing performance of Brain-Semantoks fully fine-tuned models and strong supervised baselines in Table 2 With only a linear probe, Brain- Seinn-Semnttks outperforms all baselines on eight diverse tasks. The ability to surpass fully supervised models, which are trained end-to-end on tsse-ipcc data, highlights the utility of the learned representations.
Finally, we evaluate generalization task-based fMRI using the Hariri emotion task from UKB, where participants match shapes or emotional faces in blocks. We predict block type from short seg- ments, requiring within-subjec discrimination across substantially shorter timescales than pretrain- ing sequences. We address this temporal mismatch by leveraging the masked distillation framework. Specifically, we construct a single temporal patch from a recording segment (e.g. a task block) and mask all remaining positions, thereby requiring the model to generate summary representations from limited context. We explore multiple patch construction strategies which we visualize and detail in Appendix A.7. Brain-Semantoks substantially outperforms Brain- across all settings (Table
5.2 SCALING LAWS OF SEMANTIC FMRI REPRESENTATIONS
We conducted the first detailed scaling analysis for fMRI foundation models under a linear prob- ing protocol to understand how performance varies with pretraining data size. We trained Brain- Semantoks on subsets of the UKB dataset and evaluated on both in-distribution (UKB hold-out) and out-of-distribution tasks.
8

---

Published as a conference paper at ICLR 2026
ASD
WISC
Age
MDD
ALMPA220
40-
28
22.5
Checc
250
constattoo
Centrs
Censtaation
centall
gen
Figure 3: We investigate the predictive performance of individual network representations.
C10
Jesrage Difannnc
52.1
g
Collec
53.4 30
Spalaaly Agper Apprepitnt Coar
Figure 4: Tokenizer Ablations. A) Linear probing comparison between our semantic tokenizer vs (minus) a linear projection layer applied to ROIs individually. B) We ablate aggregating into differ- ent numbers of 'networks' spatial tokens. Grey lines denote downstream tasks. Pretraining dynamics visualised by the cosine similarity between the teacher tokens and reconstructed student tokens. D) Temporal patch size ablation. TTR: Teacher-guided Temporal Regularizer
As shown in Figure performance nearly all tasks improves predictably with the logarithm of the pretraining data size, following power-law relationship characteristic of foundation models in other domains. Critically, we observe strong scaling laws for out-of-distribution (OOD) generalization. For age and sex prediction (Mttrain,prod probe = 500), which enable comparisons for matched prediction tasks, we observe consistent performance increases with more pretraining data. Remarkably, we note strong and reliable scaling for HBN, which has an age gap of more than 20 years with the UKB (HBN: up to 22, UKB: from 44 years old). Across the majority of tasks, observe plateau in OOD scaling. Yet, in contrast to fields such as language processing, downstream probing performance not only affected by scaling, but also by the baseline performance. We found that this starting point is significantly increased in Brain-Semantoks, as it outperforms ROI-level projection by up 12% when both are randomly initialized.
5.3 INTERPRETABILITY
A common challenge for interpreting deep models is that post-hoc analyses, like input masking, shift the data out of the training distribution, potentially yielding unreliable results. Our distillation pretraining with slice-masking directly addresses this. Because the model is trained to predict global information from seeing only a subset of brain networks, we can probe its learned dependencies in an "in-distribution" manner.
We assessed the importance of each of the functional networks for various downstream tasks by masking all but one network and evaluating linear probing performance (Figure 3 This reveals which individual networks contain the most predictive information for given phenotype. Multiple findings align well with neuroscientific research such the importance of the default-mode net- work for ASD and subcortical regions for MDD (Ramasubbu et al.) 2014). Interestingly, whereas the default-mode network has dominated MDD research, we find that cerebellar activity more predictive, which a more recent hypothesis (Wang et al. 2023).
ABLATIONS
We perform post-hoc hoc ablation studies on core aspects of the Seain-Semantoks framework to analyze the contribution of each component. We average linear probing performance across ten downstream

---

Published as a conference paper at ICLR 2026
Table 4: Tok-
Table Duration
TTR
Table 6: Mask- ing Type
Table 7: Loss
Table Mask- ing Ratio
enizer Conv.
Components
Kernel
Score
Duration
Score
Masking Type
Score
CLS
ATok
Score
Ratio
Score
Single Bennch
0%
50.55
Ranken
$1,03
No
1.0
47,52
[0.1.0.9)
50.33
20
0.55)
Dense
50:50
1008
49.60
Network
$2.09
Yos
0.0
50.10
00.5. 0.78]
$1.88
DW-Sr 16
$2.19
Temponal Slice
$1.50
Yes
52.39
[0.65, 0.85]
$2.39
Slice
52.39
Yes
Doal Branch
Dense 3 DW 16
$1.35
Deuse DW Str
$2.39
Ablation studies using average balanced accuracy for linear probing across ten downstream tasks. In table "Dense' and 'DW" denote standard (channel-mixing) and depthwise convolutions, re- spectively. Suffix 'Str' indicates the proposed structured kernel (as opposed to standard free-form). Integers denote kernel length. AT Weight on token masking loss. TTR Duration: active pretrain- ing period. Loss Components: First row (CLS=No) represents JEPA-style baseline.
tasks, omitting HBN-Age and HBN-Sex to reduce the influence of demographic prediction on the total score. First, we compare using linear projection layer operating on single ROIs (as in Brain- JEPA), instead of our semantic tokenizer (Figure AA). We observe unstable training dynamics, as the cosine similarity between the student's reconstructed tokens and teacher tokens (i.c., the neg- ative of the token-level loss) to quickly reach 0.95 and stabilize there (Figure CC. This indicates partial collapse where the learned representations are simple and can be predicted high accuracy immediately, which is associated with poor downstream performance. Indeed, we find large gains in downstream performance by adopting the semantic tokenizer, which also results in improved training dynamics, although instability still exists early in training. By including Teacher-guided Temporal Regularisation (TTR), which we decay to zero the first 5% training, we observe sta- ble pretraining dynamies and strong downstream performance. We find using TTR for the entirety of pretraining to overly restrictive (Table(3). We also ablate the global distillation loss (CLS Loss in Table 7 which observe to be important for downstream performance.
We furthermore ablate the choice of nine functional networks for the semantic tokenizer and we compare spatially aggregating more aggressively (into 1 spatial token per temporal patch) less so (20 or 58 spatial tokens; Figure 43; see appendix Table [3] for details). We observe significant bias for most downstream tasks towards fewer spatial tokens, with the best overall performance for the nine network solution. We also ablate the temporal patch size and convolutional filter bank for the tokenizer. finding that relatively long patches and structured kernels are important (Table 4
Next, we provide ablations on masking. We find that the influence of the mask loss should not be too high (Table 7) masking types which reduce interpolation learning are most effective (Table G). combined with high masking ratio (Table 8). Finally, we perform ablations on atlas choice (Appendix (.44T], temporal resolution (Appendix [A.4.2). augmentations (Appendix AA..3). crop
length T Toop (Appendix A.4.4), and the coding rate regularizer Rt (Appendix A.4.5)..
7 DISCUSSION
This paper presents Srain-Semantoks, novel foundation model for fMRI that marks a signif- icant shift to learning abstract, semantic representations of brain dynamics. By introducing saatosccentffcallyyggomdee
semantic tokenizer and employing a self-distillation objective, the model effectively learns high-level phenotypic signatures. The results demonstrate the strength of this approach, achieving state-of-thed performance under rigorous linear probing protocol and often surpassing supervised methods on diverse tasks. This indicates the learned representations broadly applicable without domain adaptation.
Future work may benefit from more extensive integration of task-based fMRI data and thorough investigations to understand which distribution shifts are particularly harmful for transfer. Further- more, while find that using ceuroscience-basee functional networks is effective for many down- stream tasks, follow-up research may explore learning how to group ROIS from data rather than relying on fixed mappings.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
This research was funded by Gemeinnützigen Hertie-Stiftung and the Deutsche Forschungsgemein- schaft (DFG) through FOR 5187 (project number 442075332). Additional support was provided by the Machine Excellence Cluster and DFG through the Germany's 's Excellence Strategy (EXC 2064 project number 390727645) and the following projects: CRC 1404 (project number 414984028), TRR 265 (project number 4021 (0417,,,) and RU 5363 (project number 459422098).
ETHICS STATEMENT
This research utilized publicly available. pre-existing neuroimaging datasets, including the UK Biobank, ABIDE, HBN, SRPBS, LEMON, and ADHD200. No new data were collected for this study, All data were acquired in accordance with the data use agreements specific each dataset, such as the UK Biobank application process. The original data collection procedures for these co- horts were conducted under the approval of their respective institutional review boards (IRBs) and ethics committees, with all participants providing informed consent. We have complied with all data privacy and sharing agreements, ensuring that sensitive information is stored securely and handled according to the established protocols.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our findings, we provide code and pretrained models https: 7/github. com/ SamGi jeenmiisse// Jsen/BBrain Sttt/rraii-SSSatooee
A comprehensive description of our methodology. including model architecture, training procedures, and evaluation protocols, is pro- vided in the Experimental Setup (Section 3). Specific hyperparameter configurations for all stages of our experiments are detailed in Appendix T7] All datasets used in this work are publicly ac- cessible, and relevant citations are provided to guide their acquisition. These resources are intended to allow for the full verification of our results and facilitate future research building upon our work.
LLM USAGE STATEMENT
The authors utilized large language models (LLMs) during the preparation of this manuscript. Specifically, Google's Gemini was used for proofreading, correcting grammatical errors, and im- proving sentence structure for clarity. All LLg-generated suggestions edited by the authors to ensure the final text accurately reflects our research and claims. The core scientific contributions and claims were solely authored by the human authors.
manually reviewed and
REFERENCES
Lindsay M Alexander, Jasmine Escalera, Lei Ai, Charissa Andreotti, Karina Febre, Alexander Man- gone, Natan Vega-Potler, Nicolas Langer, Alexis Alexander, Meagan Kovacs, et al. An open resource for transdiagnostic research in pediatric mental health and learning disorders. Scientific data, 4(1):1-26 2017.
Elena A Allen, Eswar Damaraju, Sergey M Plis, Erik B Erhardt, Tom Eichele, and Vince D Calhoun. Tracking whole-brain connectivity dynamics in the resting state. Cerebral cortex, 24(3):663-676, 2014.
Anahit Babayan, Miray Erbey, Deniz Kumral, Janis D Reinelt, Andrea MF Reiter, Josefin Röbbig, H Lina Schaare, Marie Uhlig, Alfred Anwander, Pierre-Louis Bazin, et al. mind-brain-body dataset of mri, eeg, cognition, emotion, and peripheral physiology in young and old adults. Sci- entific data, 6(1):1-21, 2019.
Hasan A Bedel, Irmak Sivgin, Onat Dalmaz, Salman UH Dar, and Tolga Çukur, Bolt: Fused window transformers for fmri time series analysis. Medical image analysis, 88: 102801,, 2023.
Pierre Bellec, Carlton Chu, Francois Chouinard-Decort Yassine Benhajali, Daniel S Margulies, and Cameron Craddock. The neuro bureau adhd-200 preprocessed repository. Neuroimage, 144:275-286, 2017.
11

---

Published as a conference paper at ICLR 2026
Randy L Buckner, Fenna M Krienen, Angela Castellanos, Julio C Diaz, and BT Thomas Yeo. The organization of the human cerebellum estimated by intrinsic functional connectivity. Journal of neurophysiology, 106(5):2322-238) 223422222222,,,,,, 2011.
Katherine S Button, John PA loannidis, Claire Mokrysz, Brian A Nosek, Jonathan Flint, Emma SJ Robinson, and Marcus R Munafò. Power failure: why small sample size undermines the reliability of neuroscience. Nature reviews neuroscience, 14(5):365-376, 2013.
Josue Ortega Caro, Antonio H de O Fonseca, Christopher Averill, Syed A Rizvi, Matteo Rosati, James L. Cross, Prateek Mittal, Emanuele Zappala, Daniel Levine, Rahul M Dhodapkar, et al. Brainlm: A foundation model for brain activity recordings. bioRxiv, pp. 2023-09, 2023.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties self-supervised vision transformers. Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650-9660, 2021.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning. pp. 1597-1607. PmLR, 2020.
Cameron Craddock, Yassine Benhajali, Carlton Chu, Francois Chouinard, Alan Evans, András Jakab, Budhachandra Singh Khundrakpam, John David Lewis, Qingyang Li, Michael Milham, et al. The neuro bureau preprocessing initiative: open sharing of preprocessed neuroimaging data and derivatives. Frontiers in Neuroinformatics, 7,27):5, 2013.
Dean C Delis, Edith Kaplan, Joel H Kramer, and Beth A Ober. California verbal learning test (cvlt). ECPA United States, 2008.
Adriana Di Martino, Chao-Gan Yan, Qingyang Li, Erin Denio, Francisco X Castellanos, Kaat Alaerts, Jeffrey S Anderson, Michal Assaf, Susan Y Bookheimer, Mirella Dapretto, et al. The autism brain imaging data exchange: towards large-scale evaluation of the intrinsic brain archi- tecture in autism. Molecular psychiatry, 19(6):659 667, 2014.
Zijian Dong, Ruilin Li, Yilei Wu, Thuan Tinh Nguyen, Joanna Chong, Fang Ji, Nathanael Tong, Christopher Chen, and Juan Helen Zhou. Brain-jepa: Brain dynamics foundation model with gradient positioning and spatiotemporal masking. Advances in Neural Information Processing Systems, 37:86048-86073, 2024.
Evan M Gordon, Timothy Laumann, Babatunde Adeyemo, Jeremy Huckins, William M Kelley, and Steven E Petersen. Generation and evaluation of a cortical area parcellation from resting-state state correlations. Cerebral cortex, 26(1):288-303 2016.
Lean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, et al. Bootstrap your own latent-a new approach to self-supervised Icarning. Advances in neural information processing systems, 33:21271-21288 2020.
Kuan Han, Haiguang Wen, Junxing Shi, Kun- Han Lu, Yizhen Zhang, Di Fu, and Zhongming Liu. Variational autoencoder: An unsupervised model for encoding and decoding finri activity in visual cortex. Neurolmage, 198: 136, 2019.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 9729-9738, 2020.
Xuan Kan, Wei Dai, Hejie Cui, Zilong Zhang, Ying Guo, and Carl Yang. Brain network transformer. Advances in Neural Information Processing Systems, 11125586-2559,, 2022.
Jung-Hoon Kim, Yizhen Zhang, Kuan Han, Zheyu Wen, Minkyu Choi, and Zhongming Liu. Rep- resentation learning of resting state fmri with variational autoencoder. Neurolmage, 241:118423, 2021.
Yuhong Li, Tianle Cai, Yi Zhang, Deming Chen, and Debadcepta Dey. What makes convolutional models great long sequence modeling? arXiv preprint arXiv:22 10.00998, 2022.
12

---

Published as a conference paper at ICLR 2026
Nikos K Logothetis. What do and what cannot do with finri. Nature, 453(7197):869- 2008.
Karla L Miller, Fidel Alfaro-Almagro, Neal K Bangerter, David L Thomas, Essa Yacoub, Junqian Xu, Andreas J Bartsch, Saad Jbabdi, Stamatios N Sotiropoulos, Jesper LR Andersson, et al. Mul- timodal population brain imaging in the uk biobank prospective epidemiological study. Nature neuroscience, 19(1 1031121313330 2016.
Seiji Ogawa, Tso-Ming Lec, Alan R Kay, and David W Tank. Brain magnetic resonance imaging with contrast dependent on blood oxygenation, proceedings of the National Academy of Sciences, (7(24):9868-9877 1990.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07793, 2023.
Walter HL Pinaya, Andrea Mechelli, and João R Sato. Using deep autoencoders to identify abnormal brain structural patterns in neuropsychiatric disorders: large-scale multi-sample study. Human brain mapping, 40(3):944 954, 2019.
Russell A Poldrack, Chris Baker, Joke Durnez, Krzysztof J Gorgolewski, Paul M Matthews, Mar- cus R Munafö, Thomas Nichols. Jean -aptiste Poline, Edward Vul, and Tal Yarkoni. Scanning the horizon: towards transparent and reproducible neuroimaging research. Nature reviews neuro- science, 18(2):15-126, 2017.
Rajamannar Ramasubbu, Nithya Konduru, Filomeno Cortese, Signe Bray, Ismael Gaxiola-Valdez, and Bradley Goodyear. Reduced intrinsic connectivity amygdala in adults with major depres- sive disorder. Frontiers in psychiatry, 5:17, 2014.
Ralph M Reitan. Trail making test. Reitan Neuropsychology Laboratory, 1992.
Alexander Schaefer, Kong, Evan M Gordon, Timothy O Laumann, Xi-Nian Zuo, Avram Holmes, Simon B Eickhoff, and BT Thomas Yeo. Local-global parcellation of the human cerebral cortex from intrinsic functional connectivity mri. Cerebral cortex, 28(9):3095331 14, 2018.
Eleanor Messing Semel, Elisabeth Hemmersam Wiig, Wayne Secord, al. Clinical evaluation of language fundamentals. 2003.
Oriane Siméoni, Huy Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Mare Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, et al. Dinov3. arXiv preprint arXiv:2508. 10104, 2025.
Rolf Steyer, Peter Schwenkmezger, Peter Notz, and Michael Eid. Der mehrdimensionale befindlichkeitsfraeehggnn mdbf [multidimensional mood questionnaire). Göttingen, Germany: Hogrefe, 1997.
Saori C Tanaka, Ayumu Yamashita, Noriaki Yahata, Takashi Itahashi, Giuseppe Lisi, Takashi Ya- mada, Naho Ichikawa, Masahiro Takamura, Yujiro Yoshihara, Akira Kunimatsu, al. multi- site, multi-disorder resting-state magnetic resonance image database. Scientific data, 8(1):227, 2021.
Ye Tian, Daniel S Margulies, Michael Breakspear, and Andrew Zalesky. Topographic organization of the human subcortex unveiled with functional connectivity gradients. Nature neuroscience, 23 (11):1421-1432, 2020.
Julius Vetter, Jakob H Macke, and Richard Gao. Generating realistic neurophysiological time series with denoising diffusion probabilistic models. Patterns, 5(9), 2024.
Diego Vidaurre. Stephen M Smith, and Mark W Woolrich. Brain network dynamics are hierarchi- cally organized in time, Proceedings of the National Academy of Sciences, 11((4)) 12827-12832. 2017.
13

---

Published as a conference paper at ICLR 2026
Cheng Wang, Yu Jiang, Zhihao Peng, Chenxin Li, Changbae Bang, Lin Zhao, Jinglei Lv, Jorge Sepulcre, Carl Yang, Lifang He. et al. Towards a general-purpost foundation model for fmri analysis. arXiv preprint arXiv:2506.11 11667, 2025.
Xiang Wang, Jie Xia, Weiyan Wang, Jingjie Lu, Qian Liu, Jie Fan, Tamini Soondrum, Quanhao Yu, Changlian Tan, and Xiongzhao Zhu. Disrupted functional connectivity the cerebellum with default mode and frontoparietal networks in young adults with major depressive disorder. Psychiatry research, 324:1 2023.
David Wechsler. Wechsler intelligence scale for children. 1949.
Ziyang Wu, Jingyuan Zhang. Druv Pai, XuDong Wang, Chandan Singh, Jianwei Yang. Jian- feng Gao, and Yi Ma. Simplifying dino via coding rate regularization. arXiv preprint arXiv:2502.10385 2025.
Yanwu Yang. Chenfei Ye, Guinan Su, Ziyao Zhang, Zhikai Chang, Hairui Chen, Piu Chan, Yue Yu, and Ting Ma. Brainmass: Advancing brain network analysis for diagmosis with large-scale self-supervised learning. IEEE transactions medical imaging, 43(1 1):4004 2024.
BT Thomas Yeo, Fenna M Krienen, Jorge Sepulcre, Mert R Sabuncu, Danial Lashkari, Marisa Hollinshead, Joshua Roffman, Jordan W Smoller, Lilla Zöllci, Jonathan R Polimeni, et al. The organization of the human cerebral cortex estimated by intrinsic functional connectivity. Journal of neurophysiology. 2011.
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:2111.07832, 2021.
A APPENDIX
A.1 BASELINE METHODS
Besides BrainLM and Brain-JEPA, we further compare against strong supervised approaches. The most widely-used method for prediction with fMRI is to compute pairwise Pearson corre- lations between ROI time series as measure of 'functional connectivity' (FC) and use a sup- port vector machine for prediction. We select the regularisation strength using the validation set (C =C 10-2, 10-1,11 10, 102, 10$)). Additionally, BolT is Transformer-baser architecture that classifies fMRI time series by using novel fused window attention mechanism to hierarchi- cally build representations from local to global temporal scales (Bedel et al.) (023). Next, Brain Network Transformer (BNT) utilizes ROI connection profiles for positional encoding and introduces an Orthonormal Clustering Readout mechanism. This novel pooling function learns cluster-aware embeddings by grouping functionally similar brain regions to improve graph-level predictions (Kan) eet al., 2022). Finally, BrainMass operates on functional connectivity values and uses multi-task pretraining learn latent representation robust random masking (Yang al. 2024
A.2 EXTENDED TASK INFORMATION
HBN: WISC FSIQ. The Wechsler Intelligence Scale for Children (Wechsler 1949) pro- vides broad measure for individuals cognitive ability (i.e., 'Full Scale Intelligence Quo- tient'), ) spanning verbal comprehension, visual spatial reasoning, fluid reasoning, working memory, and processing speed.
HBN: CELF. The Clinical Evaluation of Language Fundamentals test provides a measure of multi dintii-dmmesionall language performance, including language memory, language struc- ture, language context, language expressivity, and language receptivity (Semel al. 2003).
We use the total score of the scale.
LEMON: CVLT. A German adaptation of the California Verbal Learning Test Delis et al.
2008), which is a memory- based task assessing how well a person can recall and recognize list of words.
14

---

Published as a conference paper at ICLR 2026
TMT. The Trail Making Test is widely used heuropsychological test assessing cognitive functions (Reitan 1992). We use the commonly used 'B-A' score, which a dif- ference score between simpler (A) and complex (B) task, which aims to isolate executive functioning.
LEMON:
LEMON: MDBE mood state questionnaire administered German (Mehrdimensionaal Befindlichkeissfrageoggnn Steyer et al. (1997)) The three subscales (Good-Bad, Awake- Tired, Calm-Nervous) were averaged to create a composite score of general positive affec- tive state, following observation of strong inter-correlations between the components (r .51-71).
Table Summary of datasets, tasks, and sample sizes for downstream evaluations. "Tr/V/T" stands for Bainn/aliiatiin//TT
Dataset
Task
Classes
Total Size
Split Ratio (Tr/V/T)
Train Size
UKB
Sex
2
1625
0.31 / 0.08 / 0.61
500
UKB
Age
5
1625
0.31 0.08 / 0.61
500
HBN
Sex
2
1870
0.27 / 0.27 / 0.46
500
HBN
Age
5
1870
0.11 /0.11 0.78
200
HBN
WISC FSIQ
3
884
0.60 0.20 / 0.20
530
HBN
CELF Total
3
1005
0.60 0.20 / 0.20
603
LEMON
CVLT
3
212
0.60 0.20 0.20
127
LEMON
TMT B-A
3
212
0.60 / 0.20
127
LEMON
MDBF
3
213
0.60 0.20 0.20
127
SRPBS
Schizophrenia
2
291
0.60 0.20 0.20
174
SRPBS
MDD
2
499
0.60 0.20 0.20
299
ABIDE
Autism
2
974
0.60 0.20 0.20
584
A.3 TOKENIZER CONVOLUTIONS
The Conv branch uses short kernels for local, cross-ROI dependencies. Meanwhile,the ConVar branch uses longer, decaying kernels to enforce temporal inductive bias absent in standard VIT architectures. Specifically, due to the hemodynamic properties of the BOLD signal, it is both highly autocorrelated and smooth. We therefore hypothesize temporal locality prior is beneficial, com- bined with a longer kernel to model the slow-evolving signal.
To transform the input time series from C, ROIs to channels, an expansion factor is computed as H ceil D/2 ("heads'). Each ROI then processed by its set of H filters, resulting in H C, total channels. We furthermore adopt a learnable residual connection used in (Li et al. 2022 Vetter et al., 2024), which analogous to the direct feedthrough term used in state-space models. Finally, linear projection layer maps from H x Cn to the target 7 dimensionality, ensuring fixed output dimensionality while also mixing information across ROIS
Whereas the standard convolutions (Convaa) uses kernel size of 3, the structured, depthwise- separable convolutions (Conv,u) use base kernel size across 3 scales. Specifically, kernel of length 16 is created by concatenating three kernels: 4-leghhh kernel with weight 4.0, a lenghh kernel with weight 2.0, and an 8-Iength kernel (upsampled from 4 parameters) with weight 1.0.
A.4 EXTRA ABLATION EXPERIMENTS
A.4.1 ABLATIONS BRAIN ATLASES
We present ablation results on the atlas (parcellation) choices in Table 10 Overall, these abla- tions are in line with the literature in that a moderate amount of ROIs generally works well. If too few are used, the parcellation step destroys too much signal. Too many and one is liable to model too much noise. We note that for the Gordon-333 atlas (Gordon et al. 2016) map the 333 cortical ROIs to 'communities' using the provided membership: the parcellation thereby offers alternative aggregation solution the Yeo-networks. Specifically, these communities
15

---

Published as a conference paper at ICLR 2026
Allas Combination
ASD ABIDE
CELF
WISC
CVLT LEMON
MDBF LEMON
TMT LEMON
MDD
52
Age
Sex
Global
SKPBS
SRPBS
UKB
UKB
Rase (5400-T3-87)
65.13
42.18
40.87
42,10
40,23
42.88
62.40
69.26
31.15
57.52
$2.39
Cortical Parcellation Variations
8400 - $100
-1.82
-2.74
-1.49
-2.19
-.61
-3.1
+0.03
+0.78
-1.75
-800
-2.57
5400 5800
-0.28
40.56
-1.89
-2.33
-3.16
-3.07
-1.27
-2.14
-016
+0.41
-1.33
Gordon335
-2.60
Sutcortical Parcellotion Voriation
0.47
CoreBellar Porcellation Vansation
B7 + B17
-0.77
-0.04
-2.17
-0.17
-014
-0.09
Table 10: Performance comparison across atlas variations presented as average balanced accuracy (%) across three model seeds, each evaluated using 10 validtiali repetitions. Base case uses Schaefer-400, Tian-3, Buckner-7. Changes are absolute differences in balanced accuracy (percent- age points) relative to base.
emerged from applying Infomap, data-driven community detection algorithm, to parcel-to- -orrrreel connectivity data. The community names are as follows: Default, SomatomotorHand, Somatomo- torMouth, Visual, Frontoparietal, Auditory, CinguloParietal, Retrosplenia/Temporall CinguloOper- cular, Salience, Dorsal/ttention, and remaining ROIs. Interestingly, the Gordon-333 solution with total of 15 spatial tokens still yields strong downstream performance, indicating robustness the specific network mapping that is chosen.
Repetition Time
ASD ABIDE
CELF HBN
WISC HBN
CVLT LEMON
MDBF LEMON
TMT LEMON
MDD SRPBS
57:
Age
Sex UKB
Glabal
SRPBS
UKB
Base 12 sec, 0.5Hz)
65.13
42.18
40,87
42.10
40.23
42.88
62.00
09.26
31,15
51.52
52.39
Reprsitiinn Toue Marintioss
2 - 0,735 seu (1.36112)
-.15
-0.80
-2.11
+2.82
2.50
2242
0.61
-0.55
-1.19
Table 11: Performance comparison across repetition time variations. Base case uses sec (0.5 Hz). Changes are absolute differences in balanced accuracy (percentage points) relative to base. Global average computed as mean across the 10 tasks.
A.4.2 ABLATION OF TEMPORAL RESOLUTION
We present ablation results on the temporal resolution (repetition time: TR) in Table 11 For this analysis, did not resample the UKB data but kept it at its native TR of 0.735s for pretraining. All downstream data was upsampled to this resolution. To accommodate the longer sequences, we increase the patch length and crop length Terop by a factor of 0.735
2.0 We furthermore increase
the structured convolution kernel length by increasing the base kernel from 4 to 6 and the number of scales from to We observe that particularly for disease classification (ASD, MDD, SZ) performance drops are most severe. While this may relate to the nature of the downstream tasks, it likely that the relatively high TR of their datasets played significant role. Indeed, the SRPBS dataset has TRs of 2-2.5s and although ABIDE has a wider range, most are in the 2-38 range. For comparison, HBN has a TR of 0.8s, LEMON of 1.4s, and the UKB of 0.735s. The largest gains seen for the prediction of cognition (CVLT and TMT) the LEMON dataset, indicating these phenotypes may benefit from a higher temporal resolution. In sum, we believe that pretraining on resampled data (e.g. with TR of 2s) is sensible if one aims evaluate downstream data which is generally also resampled and has relatively higher native TR.
A.4.3 ABLATION OF DATA AUGMENTATIONS
To investigate the role of data augmentations during pretraining, compare physiologically- motivated augmentations against the corruption-base augmentations used in our main experiments. We test three augmentation strategies designed to mimic fMRI artifacts and noise sources:
ROI Mixing. Simulates cross-ROO signal interference (c.g., from head motion or spatial blurring) by blending signals between spatially adjacent ROIs. For fraction Prois of ROIs, we select random
16

---

Published as a conference paper at ICLR 2026
neighbor from the anatomical adjacency matrix and apply convex combination:
signal(roi] signal(roi] + (1 - a)) signal[neighbor]
where a (0..7, We test light (Prois 0.1) and heavy (Prois 0.2) variants.
Frequency Masking. Attenuates power in the typical BOLD frequency range (0.01-0.1 Hz) via
FFT. For each ROI, we mask random subset of frequency bins (with probability ~ (/(10, 0.3) per
ROI) by scaling with factors ~ (( 1.0), then reconstruct via inverse FFT. We test Smin = 0.8 (light) and min 0.6 (heavy).
Band-Specific Noise. Injects synthetic physiological noise as 1-2 sinusoidal components per ROI in the 0.01-0.1 Hz band:
noise Σ a, sin ftt + di)
(8)
where amplitudes O; ((.55, 1.5) noose frequencies f; ((0.11, 0.1), phases o; ~ ((0,, 2T), and noise ~ [0, Tmax). We test O max = 0.1 (light) and dmax 0.2 (heavy).
We pretrain models using: (1) temporal views only (no further augmentation), (2) physiologically- motivated augmentations at light and heavy intensities, and (3) vorruption-basee augmentations (as in main paper). All models are evaluated on the same 10 downstream tasks. Results are shown in Table 12
Table 12: Ablation of data augmentation strategies.
Performance reported as average accuracy
across 10 downstream tasks.
Augmentation Strategy
Avg. Accuracy (%)
Temporal views only
51.91
Physiological (light), or
51.85
Physiological (heavy), or
51.92
+ Corruption (main paper)
52.39
We find that physiologically-motivvee augmentations provide no benefit over temporal view se- lection alone, while simple corruption augmentations yield a small improvement. We hypothesize that (1) fMRI data already contains substantial physiological noise, so our cross-view distillation objective implicitly learns invariance these factors without explicit augmentation. (2) Our input data follows a conventional bandpass filter (0.01-0. 1Hz) to isolate hemodynamic responses. Con- sequently, were constrained applying Crequency-based augmentations within this signal-rich band. This likely creates a destructive trade-off: attempting to simulate spectral noise this range and introduce data diversity risks corrupting the underlying neural signal. Similarly, we excluded global spike augmentations (as an alternative to mimic head motion effects) as such artifacts are in- consistent with bandpass-filters filtered inputs. The effectiveness of coruption-bbsse augmentations may stem from their ability to increase data diversity without introducing structured artifacts that could interfere with the semantic-level representations our tokenizer targets.
Table 13: Ablation of Spatial Token Aggregation Strategies
Spatial Tokens
Aggregation Strategy
457 ROIs are aggregated into single group.
9
7 Yeo networks + subcortical group cerebellum group.
20
17 Yeo networks manually split subcortical groups cerchellum group.
58
17 Yeo networks (each split 3 times) 6 manually split subcortical groups + cerebellum group.*
We always use the Schaefer-400, Tian-3, and Buckner-7 parcellations. We used 'slice masking' except when using a single spatial token, as there it is only possible to mask temporal slices. We standardized the creation of masks as follows: masks were initially sampled on the level of the nine spatial tokens and, when using 20 or 58 spatial tokens, upsampled to the required spatial 'resolution'. Directly sampling masks with higher resolutions would have allowed for simpler interpolation solutions during pretraining, potentially biasing the analysis towards fewer spatial tokens.
Nooee The cerebellum parcellation was not split the atlas contains only 7 ROIs for this region.
17

---

Published as a conference paper at ICLR 2026
A.4.4 ABLATION ON TCROP
We perform a post-hoc analysis on the sensitivity to Teop by pretraining with crop lengths of 80, 100, and 120 timepoints and measuring average performance across our 10 downstream tasks. We observe that performance does not increase monotonically with crop length. While longer crops provide more information sample (and we indeed find single-crop prediction improves: 50.58% + 51.33% - 51.60%, while we average over the default 8 crops in Table 14). we hypothesize this offset by several factors:
Distillation task difficulty: Longer crops make the student-teacher matching task easier, reducing pressure to learn strong, generalizable representations.
Data diversity: Longer crops reduce the number of distinct crops per recording, decreasing sample-level diversity during pretraining.
Ensembling benefits: Longer crops have greater overlap, reducing the benefit of multi-crop ensembling during inference.
Our default choice of Top 100 appears represent favorable balance between information content and the above pretraining considerations. Overall, the method is reasonably robust across this range.
Timo
ASD ABIDE
CHLF
WISC
CVLT LEMON
MDRF
TMT LEMON
Sex
Glibba
SRPBS
Avg
(100)
65.13
42.18
40.87
31.15
87.52
Toop Varintios
-0.05
-0.17
0.68
0.79
2,07
-0.12
-1.36
+0,28
-033
-0.89
Table 14: Performance comparison across T Terop variations. Changes absolute differences in balanced accuracy (percentage points) relative to base. Global average computed as mean across the 10 tasks.
A.4.5 ABLATION ON CODING RATE REGULARIZER R.
Teloh
(ead
-3.0
aommuracaoo Ho-,, Sead 21
3.75
Ho Seat 31
-55
-15
Esoch
Figure 5: We observe model collapse at the start of training when the coding rate regularizer is ommitted.
A.5 OOD ROBUSTNESS ANALYSIS
We perform additional robustness analyses pertaining out-of-distribution generalization. Specifi- cally, we create data subsets based on site, MRI scanner, repetition time (TR), spatial resolution, and field-of-viee view (FOV). For both SRPBS and ABIDE perform and age prediction maximize sample size availability, We probe CLS loss without the Semantic Tokenizer (i.e., using ROI-wise linear projection). For SRPBS, the two ablated models exhibit stronger performance drops for increased TR, lowered resolution, and lowered FOV.
as well as versions trained without the global
Overall, results indicate that both components are important for consistent performance. However, it not obvious from these results that certain investigated scanning parameters particularly
18

---

Published as a conference paper at ICLR 2026
problematic for transfer learning. Although these analyses exhaustive and therefore do not preclude the possibility that other factors would provide critical insight, these findings suggest uni- form robustness improvements from the complete model. Such patterns may result from generally improved representation quality that is robust to scanning parameter variations rather than mere overfitting pretraining data characteristics.
130
ARDS
WWW
WERE FEE
Figure 6: OOD Robustness Analyses. Three model seeds are evaluated using ten cross-validation repetitions. The mean and SEM of downstream performance is plotted. We compare Brain- Semantoks (blue) to the ablated versions trained without CLS-loss (orange) or without Semantic Tokenizer (black). For reference, the data used for pretraining was acquired on a Siemens Skyra scanner with TR=0,735s, an in-plane resolution of 2.4mm, and a FOV of 21 Imm. Semantic Tok: Semantic Tokenizer.
A.6 DETAILS ON FMRI FOUNDATION MODEL COMPARISONS
To provide fair comparison between IMRI foundation models, standardized the linear probing evaluations and in doing so discovered significant effects on performance. This led to much stronger BrainLM evaluations than previously reported (Dong al. (024). First, the temporal crop win- dow size for BrainLM (2.5 minutes) is shorter than Brain-Semantoks (3.3 minutes) and BrainJEPA (almost 6 minutes). As these models are mainly used for subject-level prediction, this disadvantages models by limiting their available context. Second, we noted that BrainJEPA adopts batch normal- ization prior to the linear probing layer. We therefore standardized the evaluation for all models by (I) sampling eight cqually-spaced crops at test time and averaging their logits, and (2) adding batch normalization layer prior the linear probe layer. We found that both factors dramatically improve the linear probing performance of BrainLM (55.
To understand the surprising impact of batch normalization, we performed follow-up analysis. As such large effect likely arises from simple yet highly relevant information, we hypothesized it results from the mean BOLD signal per ROI. Specifically, BrainI application of robust scaling preserves differences in the mean signal for each ROI, a feature we have found to be highly predictive in internal analyses. To test this, we pretrained a new BrainLM model on data that was instead z- scored per ROI, which sets the mean of every ROI zero and thus removes this signal. Indeed, find this model performs poorly with linear probe, and crucially, the substantial benefit of batch normalization vanishes (Table 2).
We note that we similarly use multiple crops at test time for fine-tuning. Finally, whereas these analyses only the CLSS-token for BrainI.M, concatenate the CLS and average-pooled tokens
19

---

Published as a conference paper at ICLR 2026
for the main analyses. This standardizes the evaluation with and we observe mild further improvement (86.7%).
BrainLM Linear Probing Performance for UKB Sex Prediction.
Table 15: Effect of batch normalisation (BN) prior to the linear layer and ensembling over multiple crops.
Table 16: Ablation on data normalization. Reproduction by pretraining from scratch using per-roi Z-scoring.
Crops
BN
Accuracy (%)
Normalization
BN
Accuracy (%)
Crop
No
59.6
Robust Scaling
Yes
86.0
Crop
Yes
78.2
Per-ROI Z-scoringt
No
58.7
8 Crops
Yes
86.0
Per-ROI Z-scoring?
Yes
59.3
A.7 TASK-BASED FMRI
We provide an initial extension to task-based fMRI prediction using short sequences. The Hariri emotion task has a blocked design, in which participants need to match either shapes faces, with five blocks for each condition. We aim to decode which of these two block types is active during a short fMRI segment. Brain-JEPA and Train-SSemantoe we thus construct a single temporal patch from task. To determine which (if any) block is active at a the HRF-convolved design matrix. We consider the following three options
pretrained on long fMRI sequences and
given time point, for the construction of a single temporal patch, which we visualize in Figure] 7
Block (Pad): We select the time points for single block by thresholding the condition- specific regressor in the design matrix at 0. As this tends to result in roughly 220 seconds data, we zero-pad the data to the required patch length (s:32 and 40 seconds for Brain- JEPA and
respectively).
2 Blocks (Cont.): As the task design features two consecutive blocks for each condition twice, we sample a continuous crop of iMRI data from such a double-block period with random starting point. While this foregoes the need to pad, the data includes fixation/rest. period in the middle.
2 Blocks (Cat.): We sample twice as in the Block (Pad) method without replacement, and concatenate the two blocks. If the two blocks are too long fit in the temporal patch, both are cropped equally.
Given a single temporal patch of data, we zero-pad the temporal dimension obtain the full input shape the model expects. Next, use the model's learned mask token replace the zero- zero-padee padded data. As a mild form data augmentation, we assign the patch of real data as either the first or last patch. This is in-distribution for both models, as both are pretrained with similar temporal masks.
Finally, we note that compared to the resting-state pretraining data, the bask-based data is prepro- cessed differently. Specifically, it includes 5mm spatial filter and lacks the application of ICA+FIX for confound removal. This, in addition to the presence of raskee activity, constitutes a domain- gap despite also being UK- -i-oBib data.
20

---

Published as a conference paper at ICLR 2026
1.
Task Scan [C,T]
3.
Task Conditiona
Model Input [N,P]
Faces
Shapes
Shapes
Faces
Faces
Shapes
mesked
2. Craate singla paicch
Block (Pad)
2 Blocks (Cont)
Blocks (Cat.)
Networks
RO
Temporal Palches
Figure 7: Task-based 1MRI data preparation.
A.8 ADDITIONAL VISUALISATIONS
Centatoo
unek
Figure 8: Visualisations of learned network embeddings.
Learned Network Embeddings. We inspect the learned positional embeddings for each of the nine networks following pretraining (Figure 8). We summarize our main observations below:
The magnitude of similarity values between network embeddings are modest, indicating each network is represented distinetly.
Task-positive network cluster: The model identifies the well-established cluster between the Control network, the Dorsal Attention network, and the Salience/Ventral Attention net- work.
Sensory network segregation: The Visual network is segregated from most higher-order association networks.
Motor, subcortical. limbic pathways: Somatomotor Subcortical, Somatomotor Dorsal Attention and Salience/Ventral Attention, Limbic + Subcortical and Cerebellum.
Control-Default similarity: Surprisingly, the single strongest relationship between the Control and Default Mode networks. Canonically, these networks are often anti-correlated given their roles in external versus internal processing. However, embedding similarity does not necessitate simple co-activation. The Control network functions flexible hub that modulates both the Default Mode Network (DMN; internal) and the Dorsal Attention Network (DAN; external). The high similarities of Control-Defin and Control-DorsAttn position the Control network's 's embedding between these two major cognitive systems, reflecting its known functional topology.
21

---

Published as a conference paper at ICLR 2026
Time Domain 14 Learned Examples)
Temporal Extent (CDE)
Kerne Smouthness (KDE)
0.6
tariel
0.4
0.2
0.0
20
0.2
0.5
Support "empprr
Frequency Respanse Examples)
Temporal (KDE)
Frequency Responses
0.05
1.0
Normatoes sormaatton Meant
0.30
0.8
0.00
0.25
0.82
0.00
0.15
0.00
0.00
0.32
80
100
0.00
0.02
0.06 0.08
0.10
0.32
Frequency (180)
Dominnnt Per lod (secondi)
Fraquency [itc
Figure 9: Learned Kernels. Left: Example kernels and their frequency responses. Middle and right: we compute kernel statistics for all H C' learned kernels and plot the estimated density (KDE). Temporal Extent: We count TRs (2 seconds each) until 90% energy is reached; can be interpreted as the integration window. Kernel smoothness: We compute the standard deviation over diff(kernell. Temporal Periods: We estimate the preferred oscillation timescale by computing 1/ 1/argmax(ff). Frequency responses: The median magnitude across frequencies.
A.9. DETAILS ON HYPERPARAMETERS
Table 17: Hyperparameter settings for all experimental stages.
(a) Pre-training
(b) Fine-tuning
(c) Linear Probing
Hyperparameter
Value
Hyperparameter
Value
Hy Hyperparameter
Value
Optimizer
AdamW
Head Type
Linear Layer
Head Type
BN + Linear
Base LR
0.0007
Optimizer
AdamW
Optimizer
SGD
Epochs
100
Base LR
0.0001
Momentum
0.9
Patch Size
20
Epochs
50
Learning Rate
Fixed (best sel.)²
Crop Length
100
LR Schedule
Cosine Decay
Schedule
None
Teacher Momentum
0.99
Warmup
None
Epochs
50
Weight Decay
0.05 + 0.3
Batch Size
16
Batch Size
min(256, n//8
Batch Size
512
Weight Decay
0.05
Warmup Ratio
3% (Lincar)
LR Decay Rate
0.9
LR Schedule
Cosine Decay
Layer Scale Init
0.1
We fit linear layer for cach of the following learning rates in paral- lel and choose the best one based the validation data for test evaluation: (0.03, 0.01, 0.003, 0.001, 0.0003, 0.0001
22

---

Published as a conference paper at ICLR 2026
A.10 STATISTICAL TESTS
Table 18: Comparison of mean balanced accuracy (%) for fMRI time series foundation models using linear probes. Values shown as mean + std p-value Hoom-BBonfeme corrected, Wilcoxon signed-rank test). Best results in bold, second-best underlined. Bold p-values indicate statistical significance (p 0.05). - indicates best model.
Model
ARIDE
HBN CELF
HIN WISC
HBN Age
HRN Sex
53.84 + 0.044
42.03
16
2.12/-
1.72/ 0L004
Brain-JEAA
52.92 + 3.53 10.04
41.50
5.16/1000
38.34 3-4/00074 0.074
39.81 + 2.08/0.004
63.96 =
1 0L004
Beain Semantnks
63.13 2-14/-
42.18 2.07-
40.57 1 2437-
39.16 0.81 70.004
69.52 0.93/-
Model
UKB Age
UKB Sex
SRPBS MDD
SRPBS
BrainLM
30.16 1.41 70465
86.71 + 0.63/ 00,02
57.61 0,047
55.72 6.62 0.004
Braio-JPPA
0.004
Brain-Semantoks
= 1.15/-
+
4,79/
3.8/
Table 19: Comparison of mean balanced accuracy (%) against supervised and finetuned baselines. corrected, Wilcoxon signed-rank test). Best
Values shown as mean = std /p-value results in bold, second-best underlined. Bold p-values indicate statistical significance (p < 0.05). indicates best model.
Medel
UKB Age
URB Sex
HBN Age
IRN Ses
HBN CELF
HBN WISC
W0.03
Rutt4
43
1
Butl4
BAT
25.26
5.50 6.14
80.35
0.0.1
66014
87.67-
1.41 6914
05:22
6:91
62:46
1:78-1200
80.58 2 4-42 (0826
SeasMMe
25:51 +
09.72 =
81 40
66.99
86333
2:00 0/814
$5.00 =: 2-55 (MA14
Suaal.M
IBRA
Trals Semamnnk
21.15
$1.52 0.5227-
0.51 (EU14
69.53
1.0.37
42.18
2.50
48A7 1.2.4.7-
-Farnine
33.1
0.877--
N7.33 1 0.57//b/2
30 41
11..1.3.3 1.3.77/0085
00.31
42.99 1.347-
$0.42 L 1.51/1100
Male
EVIO
LEMON
LEMON TMY
SAPES MID
SRPHS
PC
50.49
6.32 70.211
32.25 29
6.09/0119
41.24
0.58/2554
65.12
00.30
4.65-10211
71.99 5.54
20.05.120
KNT
BADB
27 100
12/009
35.
E3/8833
38
10:109
55 103
05.
30-25/88844
braebte
6004
$1.95
/0R11
20 10-44:2
1/00/8814
60.20 2 2.30 84804
4.45 09014
70.26
45.45
15-45/48814
TeaLM
37.81
0.01 6urms
31,02
15.35
3.76//38
0.38
02.
2.70/08144
47.15/8815
Secarniic
01.33 5.774
12.84 12.00.
1.9.55/100
1.755
05 3.85//075
22.22/00445
48.36 3.357-
$5.05 = 4.055/0770
65.44
1,157-
63.0 = 2-357-
72.05 = 4.30//0556
5335
A.11 MODEL DEVELOPMENT DATA
Model development including extensive evaluations of pretraining stability was carried out using combination of the UKB dataset (non-overlapping with the holdout set used for our main analysies), subset of HBN non-ovverlapping with samples used for our main analyses), and large private dataset of resting-state fMRI. The development datasets allowed us to evaluate model transfer and architectural choices using large sample sizes prior to the experiments reported in the main paper.
23


Reviews:
[{"review_id": "dYG6sbEI1S", "reviewer": "Reviewer_mRvP", "summary": "The manuscript proposes a self-supervised foundation model for fMRI time series based on a transformer architecture. It introduces a semantic tokenizer that aggregates ROI time series into network-level tokens. The model is trained using a combination of three objectives: a global loss that minimizes the Euclidean distance between global summary tokens from different views, a network token loss that aligns masked and unmasked network tokens, and a temporal regularizer that uses time-averaged representations across each network. Experiments on several neuroimaging datasets (UK Biobank, ABIDE, HBN, SRPBS, and LEMON), show that the proposed model performs better than previous self-supervised and supervised methods.", "questions": "- The paper does not specify how checkpoints were selected, and it is unclear whether the ablation results are based on the validation set, as we can not select hyperparameters on the test set. Please clarify your checkpoint strategy.\n- It seems the authors used the wrong ICLR LaTeX template or edited it to fit more content.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The model is evaluated on multiple datasets, including UK Biobank, ABIDE, HBN, SRPBS, and LEMON, and shows consistent improvements over strong baselines such as BrainLM and Brain-JEPA.\n- The paper also includes extensive ablation studies on the effects of the semantic tokenizer design, temporal regularizer duration, masking type, loss components, and masking ratio.", "weaknesses": "- A main weakness is that all experiments rely solely on resting-state fMRI data, which limits the claim of being a true foundation model.\n- Tables 1 and 2 need to have a statistical comparison between the best model and other models. It is commonly done with a pairwise test. Then, p-values are usually corrected for multiple comparisons.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "N2MemsDzFk", "reviewer": "Reviewer_C6gR", "summary": "This paper introduces Brain-Semantoks, a self-supervised fMRI foundation model that combines a semantic tokenizer and a student–teacher self-distillation framework to learn high-level, temporally stable representations of brain dynamics. Instead of voxel- or region-level reconstruction, the model produces recording-level embeddings based on functional brain networks. The approach shows improved downstream performance across several neuroimaging datasets and phenotypic prediction tasks, such as age, sex, and ASD classification.", "questions": "1.\tHave you tested whether higher temporal resolution (beyond the 2s sampling) affects performance?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "•\tInnovative use of self-distillation and semantic tokenization to learn stable, abstract representations of brain activity.\n•\tClear performance improvements over existing foundation models (e.g., BrainLM, Brain-JEPA).\n•\tSemantic tokenizer proves particularly effective for demographic and clinical predictions (age, sex, ASD).\n•\tThe shift from low-level voxel embeddings to network-based embeddings is conceptually strong.\n•\tSignificant ablation studies to explore the benefit of each component into the model, which shows robust evaluations by the authors.", "weaknesses": "•\tWhile results are solid, gains from other architectural components beyond the tokenizer are more modest.\n•\tThe paper could discuss temporal resolution more thoroughly — would finer sampling (e.g., sub-2s TR) lead to better representations or unnecessary noise amplification?\n•\tThe work focuses exclusively on resting-state data; some commentary on potential extension to task-based fMRI would strengthen the contribution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1TV7Cwez6l", "reviewer": "Reviewer_sTj7", "summary": "The paper proposes Brain‑Semantoks, a self‑supervised fMRI foundation model that shifts from reconstruction to abstraction by (1) introducing a semantic tokenizer that aggregates ROI time series into network‑level tokens and (2) pretraining via self‑distillation across long temporal views with a Teacher‑guided Temporal Regularizer (TTR) for stability on noisy fMRI signals. The model shows strong linear‑probe performance across nine downstream datasets/tasks (ABIDE, HBN, UKB, SRPBS), outperforming BrainLM and Brain‑JEPA and often rivaling or surpassing fully supervised/fine‑tuned baselines, with ablations and scaling analyses.", "questions": "- Functional networks: Reducing the complexity of the input data via the use of an atlas for grouping based on functional network annotation is a sensible choice. However, as briefly discussed by the authors, there are several alternatives for such atlases (eg, Schaefer, Shen or even Yeo-17 [see ref1 for other atlases]), from which Yeo-7, in addition to the two subcortical networks, is the most minimalist. How sensitive are the results to the chosen atlas and the definition of subcortical/cerebellar groupings? Have you evaluated alternative parcellations? Have you considered a data‑driven network discovery for an appropriate number of functional networks?\n\n- Positional and network embeddings: Does the model learn network identity embeddings that capture known functional relationships? Any probing of these embeddings’ geometry vs canonical network hierarchies?\n\n- Structured convolutions: What precise kernel sequences and decay profiles were used, and how do they relate to expected hemodynamic time constants? Can you share learned filters or frequency responses to justify the inductive bias?\n\n- Coding‑rate regularizer: how is R scaled with batch size and feature dimension; what is the effect of batch‑dependent normalization on R?\n\n- Augmentations: The paper uses light corruptions (channel/time zeroing, Gaussian noise, amplitude scaling), which likely result in out-of-distribution data. Have you tested physiological noise models or band‑limited perturbations that mimic TR/resampling artifacts?\n\nReferences:\n\n[ref1] https://www.lead-dbs.org/helpsupport/knowledge-base/atlasesresources/cortical-atlas-parcellations-mni-space/", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Clear reframing toward semantic abstraction with a neuroscience‑grounded tokenizer operating at functional network granularity, reducing token length and noise while injecting inductive bias.\n\n- The slice masking to avoid trivial interpolation is a strong regularization that forces the model to learn meaningful relationships between tokens. \n\n- Well‑designed curriculum via TTR that averages network tokens over time early in training, improving stability of the model during training\n\n- Rigorous linear‑probe protocol with 10‑fold CV and standardized heads; strong results across clinical and demographic tasks\n\n- Detailed scaling analysis for fMRI FMs under linear probing, showing power‑law like gains with pretraining size and OOD improvements without domain adaptation.", "weaknesses": "- The atlas choice is mostly arbitrary. No analysis of how results change with alternative parcellations (Schaefer, Shen, Yeo‑17) or different subcortical/cerebellar groupings; no exploration of data‑driven network discovery to justify the choice of nine functional networks.\n\n- The geometry of learned network identity embeddings is not analyzed; it is unclear whether they capture canonical inter‑network relationships or known hierarchies.\n\n- Precise kernel sequences and decay parameters are unclear; there is no connection to hemodynamic time constants nor visualization of learned filters or frequency responses to support the inductive bias.\n\n- The scaling of the coding‑rate regularizer with batch size and feature dimension is not described; the influence of batch‑dependent normalization on the covariance and the regularizer is not evaluated.\n\n- There is no evaluation of physiologically realistic, band‑limited, or acquisition‑mimicking perturbations (e.g., motion, respiratory/cardiac noise, TR/resampling artifacts) beyond simple channel/time zeroing, Gaussian noise, and amplitude scaling.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jl3JKxXERe", "reviewer": "Reviewer_TTS1", "summary": "This paper presents Brain-Semantoks, a novel fMRI foundation model that shifts the paradigm from reconstruction-based learning to abstraction-based representation learning. The authors propose a neuroscientifically grounded semantic tokenizer that aggregates voxel signals into functionally meaningful network-level tokens, coupled with a self-distillation objective across temporal views to learn stable, high-level representations of brain dynamics. To mitigate instability from noisy fMRI data, a Teacher-Guided Temporal Regularizer (TTR) is introduced to gradually guide the model toward robust convergence. Extensive experiments on multiple public datasets show that Brain-Semantoks achieves state-of-the-art linear probing performance, outperforming both supervised and existing self-supervised methods, with strong generalization across tasks without domain adaptation. Overall, the work makes a compelling case for semantically informed, abstract representation learning in neuroimaging foundation models.", "questions": "1. Can you report adapter-based or LoRA-style finetuning on a subset of downstream tasks to test whether the learned features remain robust under end-to-end adaptation and to compare against masked-JEPA/BrainLM finetuned baselines?\n\n2. How do results vary across alternative parcellations and numbers of networks/tokens (e.g., Schaefer 100/400, Glasser 360)? You ablate 1/9/20/58 tokens—could you include cross-parcellation comparisons and per-task preferences? \n\n3. Please provide per-site/scanner/TR robustness analyses and/or harmonization baselines (e.g., ComBat) to substantiate OOD claims; which specific shifts most degrade transfer, and does the semantic tokenizer mitigate them? \n\n4. Could you include a matched JEPA/MAE-style latent reconstruction baseline with the same tokenizer and encoder to isolate the benefit of self-distillation + TTR? \n\n5.  Beyond cropping two long segments, did you test time-warping/jittering or frequency-domain perturbations that preserve physiological plausibility? How sensitive is performance to Tcrop? \n\n6. Any early evidence on task-fMRI or clinical endpoints (diagnosis/prognosis) to complement resting-state tasks, even if small-scale?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1.  Reorients fMRI foundation modeling from masked reconstruction to semantic abstraction via a functional-network tokenizer, aligning tokenization with neuroscientific priors rather than voxel/ROI grain. The self-distillation across temporal crops plus TTR curriculum is a thoughtful adaptation of DINO/iBOT-style ideas to low-SNR fMRI. \n\n2. Solid ablation suite quantifies the effect of tokenizer design (structured conv, patch length), masking scheme/ratio, and TTR duration; results cohere with the hypotheses (e.g., higher masking ratios; network/slice masking reducing interpolation; moderate mask-loss weight). \n\n3. Clear articulation of the problem with reconstruction-centric objectives for phenotype prediction and a well-motivated methodology figure/flow (student-teacher; temporal views; semantic tokens). Related work positioning is balanced. \n\n4. Demonstrates SOTA linear-probe performance across diverse tasks, often surpassing supervised methods, suggesting practically valuable, broadly transferable embeddings; claims of scaling and OOD gains are timely for neuro-foundation models.", "weaknesses": "1. Heavy emphasis on linear probing; fewer results on full/partial fine-tuning (or lightweight adapters) and limited analysis of task-based fMRI limits claims about universal transfer beyond resting-state embeddings. The authors acknowledge the task-fMRI gap as future work. Including even a small task-fMRI probe or adapter-tuning study would strengthen external validity. \n\n2. Tokenizer depends on fixed functional networks; while neuro-plausible, robustness to alternative parcellations or data-driven grouping is only proposed as future work. A sensitivity analysis across multiple parcellations (e.g., Schaefer/Glasser granularity) would bolster generality. \n\n3. Cross-dataset variability (hardware/protocols/cohorts) is central; yet the paper could provide a deeper distribution-shift audit (e.g., per-site/per-scanner breakdowns, ComBat-style harmonization baselines, robustness to TR differences) to substantiate OOD claims. Authors note investigating harmful shifts as future work. \n\n4. While ablations are thorough, a targeted component attribution (e.g., semantic tokenizer vs. self-distillation vs. TTR via controlled swaps against strong masked-JEPA baselines) would clarify which element drives most of the gain. Current tables hint at trends but don’t isolate effect sizes against matched alternatives. \n\n5. The “rigorous linear-probe protocol” could use a compact table specifying classifier type, search space, train/val splits, number of repeats, and CIs across all tasks to help reproducibility reviewers quickly verify comparability. (Some of this is in appendices; mirroring a summary in the main text would help.)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)