BrainLM: A foundation model for brain activity recordings
Authors: Josue Ortega Caro
Antonio Henrique de Oliveira Fonseca
Syed A Rizvi
Matteo Rosati
Christopher Averill
James L Cross
Prateek Mittal
Emanuele Zappala
Rahul Madhav Dhodapkar
Chadi Abdallah
David van Dijk
Venue: ICLR 2024 poster (2024)
Keywords: foundation model
fMRI
Primary area: applications to neuroscience & cognitive science
TL;DR: Trained an masked autoencoder on the largest fMRI dataset

Abstract: We introduce the Brain Language Model (BrainLM), a foundation model for brain activity dynamics trained on 6,700 hours of fMRI recordings. Utilizing self-supervised masked-prediction training, BrainLM demonstrates proficiency in both fine-tuning and zero-shot inference tasks. Fine-tuning allows for the accurate prediction of clinical variables like age, anxiety, and PTSD as well as forecasting of future brain states. Critically, the model generalizes well to entirely new external cohorts not seen during training. In zero-shot inference mode, BrainLM can identify intrinsic functional networks directly from raw fMRI data without any network-based supervision during training. The model also generates interpretable latent representations that reveal relationships between brain activity patterns and cognitive states. Overall, BrainLM offers a versatile and interpretable framework for elucidating the complex spatiotemporal dynamics of human brain activity. It serves as a powerful "lens" through which massive repositories of fMRI data can be analyzed in new ways, enabling more effective interpretation and utilization at scale. The work demonstrates the potential of foundation models to advance computational neuroscience research.

Published as a conference paper at ICLR 2024
BRAINLM:
A FOUNDATION MODEL FOR BRAIN ACTIVITY RECORD-
INGS
Josue Ortega Caro*11 Antonio H. de O. Fonseca" Syed A. Rizvi*A, *A Matteo Rosati*.. Christopher Averill?, James L. Cross*, Prateek Mittal', Emanuele Zappala") Rahul M. Dhodapkar', Chadi G. Abdallah David van Dijk
Wu Tsai Institute, 'Interdepartmentall Neuroscience Program, Yale University, BBaylor College of Medicine, Department of Computer Science, 5 Yale School of Medicine," University of Southern California, Interdepartmental Program in Computational Biology & Bioinformatics, Internal Medicine, 10 Cardiovascular Research Center, Yale University." Department of Mathematics and Statistics, Idaho State University, Co-first authors, Co-Corresponding authors
ABSTRACT
We introduce the Brain Language Model (BrainLM), foundation model for brain activity dynamics trained on 6,700 hours of fMRI recordings. Utiliz- ing self-supervised masked- prredicionn training, BrainLM demonstrates profi- ciency in both fine-tuning and zero-shot inference tasks. Fine-tuning allows for the accurate prediction of clinical variables like age, anxiety, and PTSD as well as forecasting of future brain states. Critically, the model generalizes well to entirely new external cohorts not seen during training. In zero-shot inference mode, BrainI.M can identify intrinsic functional networks directly from raw fMRI data without any network- based supervision during training. The model also generates interpretable latent representations that reveal relationships be- tween brain activity patterns and cognitive states. Overall, BrainLM offers a versatile and interpretable framework for elucidating the complex spatiotem- poral dynamics of human brain activity. It serves as a powerful "lens" through which massive repositories of fMRI data can be analyzed in ways, enabling more effective interpretation and utilization at scale. The work demonstrates the potential of foundation models advance computational neuroscience search.
INTRODUCTION
Understanding how cognition and behavior arise from brain activity stands as one of the fun- damental challenges in neuroscience research today. Functional magnetic resonance imaging (IMRI) has emerged as a critical tool for pursuing this goal by providing a noninvasive window into the working brain. fMRI measures blood oxygen level fluctuations that reflect regional neural activity. However, analyzing the massive, dingh--iiesional recordings produced by fMRI poses major challenges. The blood-oxygen-ler oxygen-level dependent (BOLD) signals represent an indirect mea- sure of brain function and can be difficult to interpret. Furthermore, fMRI data exhibits complex spatiotemporal dynamics, with critical dependencies across both space and time. Most existing analysis approaches fail to fully model these complex nonlinear interactions within and across recordings Seewoo et al. 2021).
Prior [MRI analysis techniques have relied heavily machine learning models designed for spe- cific narrow tasks (Takagi Nishimoto 2023 Mozafari et al. 2020 Ozcelik et al. 2022), hindering their generalizability. Traditional models also struggle to integrate information across the wealth of unlabeled fMRI data available. Hence, there is an ongoing need for flexible modeling frame- works that truly capitalize on the scale and complexity of fMRI repositories.

---

Published as a conference paper at ICLR 2024
Input data
Zero-shot inference
Latent Space
Atiantoon vindiiaaion
27 recoringe
BrainLM foundation model
Fine- tre-tuning tasks
Clinical vartables
Predicting cinical vatisbles
41 pes individuals (LK Bicbume Humen Connectiome Project)
Predicting activity
ansorator
Figure I: Overview of the BrainIM framework. The model was pretrained on 77,298 fMRI sam- ples, summing to 6,700 hours of fMRI recordings. BrainLM pretrained via spatiotemporal masking and reconstruction. After pretraining, BrainLM supports diverse capabilities through fine-tuning and zero-shot inference. Fine-tuning tasks demonstrate the prediction of future brain states and clinical variables from recordings. Zero- shot applicaioon include inferring func- tional brain networks from attention weights. This highlights BrainLM's versatility as a founda- tion model for fMRI analysis.
Foundation models represent a new paradigm in artificial intelligence, shifting from narrow, task-specific training to more general and adaptable models (Brown et al. (020). Inspired by breakthroughs in natural language processing, the foundation model approach trains versatile models on broad data at scale, enabling wide range of downstream capabilities via transfer learning- Unlike previous systems designed for singular functions, foundation models ex- hibit general computational abilities that make them suitable for myriad real-world applications. Large language models like GPT have demonstrated the potential of this framework across di- verse domains including healthcare, education, robotics, and more (Bommasani et al. 2021] Wiggins & Tejani 2022 Orr et al. 2022 Mai et al. 2023). Foundation models offer new oppor- tunities to rethink challenges in neuroscience and medical imaging analysis.
Here, we introduce BrainLM, the first foundation model for fMRI recordings. BrainLM leverage a Transformer-bast architecture to capture the spatiotemporal dynamics inherent in large-scale brain activity data. Pretraining on a massive corpus of raw fMRI recordings enables unsuper- vised representation learning without task-specific constraints (see Figure D After pretraining, BrainLM supports diverse downstream applications via fine-tuning and zero-shot inference. We demonstrate BrainLM's capabilities key tasks including prediction of future brain states, de- coding cognitive variables, and discovery of functional networks. Together, these results high- light BrainLM's proficiency both zero-shot and fine-tuning tasks. This work highlights the potential of applying large language models to advance neuroscience research. BrainLM is foundation model for the community to build upon, providing more powerful computational tools to elucidate the intricate workings of the human brain.
2 RELATED WORK
Prior work has explored various machine-learning techniques for analyzing fMRI recordings. Farlier approaches focused on decoding cognitive states from activity patterns. Methods like SVM and neural networks were trained in supervised fashion to classify fMRI data into stimu- lus categories or regress against variables interest (Horikawa Kamitani 2017 Hoetle et al.) 2018 Beliy et al. 2019). However, these models learn representations tailored to specific tasks and struggle
eeggleete to generalize.
Recent work has aimed to obtain more transferable fMRI encodings without task-specific con- straints. Techniques include training autoencoders to reconstruct recordings, learning to map recordings to dover-dimensional space (Takagi & Nishimoto 2023 Mozafari et al. 2020 Ozce-
2

---

Published as a conference paper at ICLR 2024
lik et al. 2022 However, most methods operate small datasets, limiting their ability learn robust generalizable representations. Our work is most closely related to recent efforts to apply masked autoencoders for unsupervised pretraining
fMRI data (Chen et al. 2023a b) or other
neural recording modalities (7appala et al. 2022 Fonseca et al. 2023 Ye & Pandarinath 2021;
Kostas et al. 2021 However, these prior papers have focused on using the autoencoder pretrain- ing only for visual cortex, and were trained on 1-2 orders of magnitude fewer data compared to
BrainLM. Furthermore, all other fMRI models have only focused on applying the MAE framework for specific applications such as stimuli reconstruction. In contrast, BrainLM is foundational model of all brain regions and has been trained on substantially more data which allows it to learn more powerful encodings of spatiotemporal IMRI patterns.
In addition, other work has focused on finding representational similarities between large lan- guage models and brain recordings (Caucheteux et al. 2022 Pasquiou et al. (022). Work along this line has found high correlation between LLMs and specific brain areas such as language
processing regions. However, this work does not focus on learning foundation models of brain dynamics nor finetuning their models for downstream biological tasks. These two are key differ- ences between their approach and ours.
3 METHODS
3.1 DATASETS AND PREPROCESSING
We leveraged two large-scale publicly available datasets the UK Biobank (UKB) MMiller al. 2016) and the Human Connectome Project (HCP) (Elam et al. 2021). The UKB comprises a robust collection of 76,296 task-based and resting-state functional MRI (fMRI recordings, accompanied by medical records, from a demographic spanning ages 40 to 69. Recordings were acquired on Siemens 3T scanner at 0.735s temporal resolution. The HCP contains 1,002 high high-quality fMRI recordings from healthy adults scanned at 0.72s resolution.
Our model was trained on 80% of the UKB dataset (61,038 recordings) and evaluated on the held-out 20% and the full HCP dataset. All recordings underwent standard preprocessing in- cluding motion correction, normalization, temporal filtering, and ICA denoising to prepare the data (Salimi- Khorshiii et al. 2014: Abdallah 2021). To extract parcel-wise time series, we par- cellated the brain into 424 regions using the AAL-424 atlas (Nemati et al. 2020). This yielded 424-dimensional scan sequences sampled at ~ Hz. Robust scaling was applied by subtracting the median and dividing by the interquartile range computed across subjects for each parcel. In total, our training dataset comprised 6,700 hours of preprocessed fMRI activity patterns from 77,298 recordings across the two repositories. This aarge-scale corpus enabled unsupervised pre- training of BrainLM to learn robust functional representations.
3.2 MODEL ARCHITECTURE & TRAINING PROCEDURE
Our model is based on a Transformer masked autoencoder structure, drawing inspiration from natural language processing designs like BERT (Devlin et al. 2018) and Vision Transformer Doosoviskki et al. 2020 He et al. 2022). At its core, the model is a series of multi-heades self- attention layers that process visible (unmasked) as well masked patches. The goal is to predict the original signal of the masked patches (refer to Figure 2 Detailed implementation specifics are elaborated upon in the supplementary materials.
During training, we selected random subsequences spanning 200 timesteps from each fMRI recording. These parcel time series were then dissected into blocks of 20 timesteps, leading to 10 von-overlapping segments per subsequence. These segments were transformed into 512- dimensional vectors, and a masking operation was performed at rates of 20%, 75%, or 90%. To facilitate model scaling to hundreds of millions of parameters, we design an alternative mask- ing scheme directly inspired by patch- image encoding in Computer Vision. We select a random subsequence of 200 timesteps from each IMRI recording, and treat the selected record- ing window of 424 parcels and 200 timesteps as a 2- 2-dimensional image, where the 424 parcels are ordered by their Y-coordinate in 3D space. This scheme preserves locality of brain regions and allows for multi multi-parcel encoding of the fMRI recording which reduces the total amount of tokens. Note that parcels tend be correlated in space, therefore, predicting masked parcels
3

---

Published as a conference paper at ICLR 2024
Reconstruction loss
Input data
Prediction
Parce's
3
N
N
Parcel 3D sportal embedding Duyall
Temporal
embestiing It]
Rendom
Trandforme Enceder
Tandormer Dwrodd
Prediction
Perceb
Patcels
Fulure mesking
Time
Uata
Encocing
Prediction
Figure 2: BrainLM architecture and training procedure. A) The fMRI recordings compressed into 424 dimensions (parcels) (See Methods 3.1. The recordings are randomly trimmed to 200 time points. For each parcel, the temporal signal is split into patches of 20 time points each (blue dashed boxes). The resulting 4240 patches are converted into tokens via a learnable linear pro- jection. B) From the total number of tokens (blue), a subset is masked (red), either randomly or at future timepoints. We then add the learnable spatial and temporal embeddings to each token. These visible tokens (blue) are then processed by a series of Transformer blocks (Encoder). The input to the Decoder is the full set of tokens, consisting of encoded visible tokens (green) and masked tokens (red). The Decoder also consists of Transformer blocks and ultimately projects the tokens back to data space. Finally, we compute the reconstruction loss between the predic- tion (purple) and the original input data (blue).
within physical region makes the task significantly more difficult compared to the single-parcel encoding. Only the segments that remained unmasked were processed by Transformer en- coder, which consists of self-attention layers and heads. This was followed by 2-layer Trans- former decoder that processed both the masked and unmasked vectors. The training regimen involved batches of 512 samples, optimized via the Adam algorithm across a span of 100 epochs. The optimization goal was to minimize the mean squared error between the original and the constructed signals (visualized in Figure 2 This pretrained model was subsequently leveraged for several tasks including zero-shot brain network inference, fine-tuning for clinical variable prediction, and prognostication of future time states.
3.3 CLINICAL VARIABLE PREDICTION
To adapt BrainLM for predicting clinical variables from IMRI recordings, we augmented the pre- trained encoder with 3-layer MLP head. This was then trained to regress targets such as age, neuroticism, PTSD, and anxiety disorder scores. We used Z-score normalization for age. Neu- roticism scores were adjusted through min-max scaling, aligning the distribution within the (0, 1] range. For both Post Traumatic Stress Disorder (PCL-5) and General Anxiety Disorder (GAD-7) scores, we first applied a log transformation to moderate their exponential distribution. This was followed by min-max scaling.
We conducted regression on clinical variables using data that BrainLM had not encountered during fine-tuning. For this, we reserved a subset of samples from the UKB test set for both fine-tuning and training SVM regressors. Our evaluation compared the performance of BrainLM with baseline models, using both raw input data and pretrained embeddings. To mitigate over- fitting during the tunittning process, introduced a 10% dropout to the activations of both the BrainLM encoder and its MLP head. We also evaluated the pretrained models on zeroshot regression using this same sample subset.
4

---

Published as a conference paper at ICLR 2024
Parcel Parcel Parcel 4
Biobank (unseen data)
BrainLM
Data
Time
Parcel 8
Human Connectome Project (external validation)
Percel
Parcel C
Time
Figure 3: BrainLM reconstruction performance on held-out data. The model predictions (red) closely fit the ground truth recordings (black) of unseen data sampled from the cohort that the model was trained on (A, UKB) as well as data sampled from an external never- lever-before-sent co- hort (B, HCP). This demonstrates BrainLM's ability to generalize across subjects and datasets.
RESULTS
MODEL GENERALIZATION
To evaluate BrainLM's ability to generalize to fMRI data, tested its performance on held- out recordings from both the UKB test set as well as the independent HCP dataset.
On the UKB test data, BrainLM achieved average R² score 0.464 on the prediction of masked patches, indicating strong generalization on unseen recordings from the same distribution. Crit- ically, BrainLM also generalized well the HCP dataset, achieving R² score of 0.278. Despite differences cohort and acquisition details, BrainI. could effectively model brain dynamics in this entirely new distribution. Figure shows sample reconstructions on both UKB and HCP recordings. We found that the model generalization scales with data size and model size, show- ing strong model scaling (see Figure Overall, these results demonstrate BrainLM's ability to learn robust representations of fMRI recordings that generalize across datasets. This highlights the advantages of large-scale pretraining for learning widely applicable models of brain activity. We will make all pretrained models available via Hugging Face.
4.2 PREDICTION OF CLINICAL VARIABLES
One of the primary advantages of foundation models lies in their capability to fine-tune for spe- cific downstream tasks, capitalizing on pretrained representations. An examination of the latent space reveals that our pretrained BrainLM model adeptly encodes information crucial the clin- ical variables tied to fMRI recordings, as showcased in TableI1ann Figure To delve deeper into BrainIM's clinical predictive capacity, we adapted the model regress metadata variables from the UKB dataset. This involved enhancing the pretrained encoder with an MLP head and fine- tuning it to forecast variables such as age, neuroticism, PTSD, and anxiety disorder scores. For this tine-tuning stage, utilized subset of the UKB samples that had been set aside and were untouched during the training process. We then compared BrainLM's performance against other prevalent methodologies used in predicting clinical variables, referencing studies like Drysdale let al. (2017); lidaka (2015). For a benchmark, we factored in the results from an SVM trained raw data predict these clinical outcomes. Impressively, across all the measured variables,
5

---

Published as a conference paper at ICLR 2024
0.25
Reconstruction R2
0.15
0.10
0.05
650M parameters
111M parameters
13M parameters
0.00
0.1
1.0
10.0
100.0
Training data (%)
Figure 4: Reconstruction performance on masked fMRI recording patches at varying amounts of training data and model parameter sizes. Increasing model size and data scale consistently yields better performance at self-supervised reconstruction of signal information.
Gender
Age
Resting state
Resting
Feralc
Mole
80
UMA0
Age
UMAP1
UMAP1
Figure 5: BrainLM learns latent space encoding clinically relevant information from fMRI recordings. For each fMRI, the CLS token extracted as summary representation. The col- lection of CLS tokens is then embedded into a 2D space using UMAP. The resulting embedding demonstrates organization based on gender (left) and age (right) of the subjects.
BrainLM consistently outperformed other methods, registering a notably lower mean squared error than the competing approaches, as detailed in Table [ Additional gains from fine-tuning further demonstrate the benefits of initializing with a meaningfully pretrained model before tun- ing to specific prediction task. We further explored the quality of the models' recording embed- dings by assessing their zeroshot metadata regression performance. seen Supp. Table 17 performance scales with model size, indicating that larger models learn more clinically signih- cant information during pretraining.
Overall, these results validate BrainLM's ability to uncover predictive signals within complex fMRI recordings. By leveraging large-scale pretraining and transfer learning, BrainLM offers a powerful framework for fMRI-based assessment of cognitive health and neural disorders. Ongoing work exploring clinical prediction across broader range of psychiatric, neurological, and neurode- generative conditions.
4.3 PREDICTION OF FUTURE BRAIN STATES
To evaluate whether BrainLM can capture spatiotemporal dynamics, we assessed its perfor- mance in extrapolating future brain states. A subset of the UKB data used to fine-tune the model to predict parcel activities at future time points. During fine- tne-tuning, BrainLM was given 180 time-step sequences and trained to forecast the subsequent 20 time steps. We compared

---

Published as a conference paper at ICLR 2024
Table Results for the regression of the clinical variables. LSTM, GCN Kipf & Welling 2016 and BrainLM were trained with fMRI recordings as input. SVR and MLP used the correlation matrix of parcels input into the model (see Drysdale al. 2017 and lidaka 2015 for more details). The values show the MSE (mean + std).
Age
PISD (PCI-5)
Anxiety (GAD-7)
Neuroticism
Raw data
2.0 0.2219
0.034 1 0.0027
0.172 0.0066
0.160 = 0.0137
SVR
0.659 +: 0.036
0.022 0.004
0.090 + 0.010
0.087 1 0.008
MLP
0.693 0.001
0.021 0.0003
0.081 1 0.001
0.079 0.0005
LSTM
0.596 0.040
0.019 0.001
0.083 0.0022
0.076 + 0.002
GCN
0.862 0.09
0.021 0.002
0.083 + 0.02
0.077 =
BrainLM 13M
0.464 + 0.0252
0.018 = 0.0008
0.074 0.0053
0.072 + 0.0049
BrainLM 111M
0.503 = 0.0207
0.015 + 0.0003
0.073 0.0031
0.069 + 0.0038
Extrapolation the UKB
2.0
BrainL M
1.5
MSE
LSTM
1.0
LatentODE
111
111
11
NODE
0.0
4
4
16
77
18
10
Extrapolated point
Extrapolation for HCP
MSE
1.0
0.5
111
E11
0.0
4
0
45
16
1
Extrapolated point
Figure 6: BrainLM outperforms other models in extrapolating future brain states. Models were trained to predict parcel activity 20 timesteps beyond observed context data. The plot shows the mean squared error per timestep held-out UKB and HCP recordings. BrainIM demonstrates significantly lower error in forecasting near- future brain states. This highlights how pretraining enables BrainLM effectively learn spatiotemporal fMRI dynamics. The time points for which BrainLM has significantly (p < 0.05) lower error than the other methods are identified with "*"
against baseline models including LSTMs (Hochreiter & Schmidhuber 1997) NODE Chen al. 2018 Rubanova et al. 2019), and non- non-pretrained version of BrainIM.
As depicted in Figure 6 the tuned BrainLM model significantly surpassed other methodolo- gies when predicting future activity, demonstrating superior performance on both the UKB and HCP test sets. The version of BrainI.M without pretraining exhibited noticeable dip in perfor- mance, underscoring the value of pretraining for such tasks. The optimized BrainLM consistently and significantly recorded the least error across all predicted timesteps for the UKB data. For HCP this was significant for the initial 8 timesteps, as detailed in Figure 6 This highlights BrainLM's robust capability to intuitively grasp the dynamics of fMRI.
4.4 INTERPRETABILITY VIA ATTENTION ANALYSIS
interpretability. Through the visualization of sell-attention weights, we glean deeper insights into the model's internal representations. We calculated the average attention the BrainLM's CLS token allocated to each parcel during the encoding of fMRI recordings. As highlighted in Figure 7 task recordings demonstrated pronounced focus on the visual cortex compared to the resting state. This aligns seamlessly with the visual stimuli introduced during tasks. Furthermore, observed distinctions in attention patterns when
key feature of BrainLM
7

---

Published as a conference paper at ICLR 2024
Table 2: Quantitative evaluation extrapolation performance. Models were tasked with fore- casting parcel activity 20 timesteps beyond observed data from the UKB dataset. We compare BrainLM NODE (Chen al. 2018, LatentODE (Rubanova
2019 and LSTM. BrainLM shows the best performance across all metrics: higher (R") and Pearson correlation coefficients (R), and lower mean squared error (MSE) between predicted and true future states. f MSES for the larger MrainLM models are not comparable to other models given different normalization of recordings.
UKB
HCP
R²
MSE
R²
R
MSE
BrainLM 650M (fine-tuned)
0.098
0.313
0.019t
0.061
0.018f
BrainLM 111M tine-tunedl
0.095
0.309
0.020f
0.056
0.244
0.0181
BrainI.M 13M (fine-tuned)
0.086
0.280
0.648
0.028
0.185
0.568
Transformer 13M [w/o pre-training)
0.012
0.112
0.695
0.007
0.090
0.583
LSTM
-0.001
0.151
0.704
-0.020
0.049
0.598
Neural ODE
-0.577
1.083
-0.469
2.010e-4
0.857
Latent ODE
0.001
0.023
0.703
-0.003
2.0026e-4
0,588
A
Difference (Task Resting
Task
Resting
depression severityye 10)
High depression severity 10)
thigh low)
Ventral Sallence
Donsel
Central Exccutive
Defruh Roce
Supcortical
Figure 7: BrainLM attention maps reveal functional contrasts. A) Differences in parcel attention weights between task (left) and rest (right), and low and high depression scores (PHQ9). Task vs. resting state differences highlight changes in the visual cortex. Comparing the severity of depres- sion, the difference highlights subcortical areas. Attention can localize parcels 7 functional networks without supervision. This demonstrates BrainI.'s ability to learn meaningful repre- sentations an unsupervised manner.
comparing mild to severe depression, as indicated by the PHQ-9 scores. Notably, there was a pronounced emphasis the frontal and limbic regions for severe depression cases in contrast to milder ones. Such attention distribution patterns resonate with an established understanding of functional shifts associated with depression, Whlloon-Wilsonn et al. 2000 Lai et al. 2000). In sum, the attention maps acquired underscore BrainLM's adeptness detecting clinically pertinent variations within functional networks. By illuminating the regions prioritized during the encoding process, the attention analysis enriches our interpretative capabilities, offering valuable neuroscientific insights. For more comprehensive view of other clinical variables, refer to the supplementary Figure
referenced in (Pizzagalli & Roberts 2022;
4.5 FUNCTIONAL NETWORK PREDICTION
We evaluated BrainLM's ability segment parcels into intrinsic functional brain networks di- rectly from fMRI activity patterns, without any network- based supervision. We organized parcels into 7 functional categories delineated in previous cortical parcellation research. These groups
8

---

Published as a conference paper at ICLR 2024
Table Comparing methods for functional region identification. Parcels from 1000 UKB record- ings were categorized into 7 regions without supervision. A kNN classifier on BrainLM's self- attention maps achieved the highest accuracy, outperforming alternatives using raw data and other representation learning techniques.
Accuracy (%)
BrainI.M (attention weights)
58.8
Raw Data
39.2
Variational Autoencoder (Kingma Welling 2013
49.4
GCN
Kipf & Welling 2016
25.9
encompassed visual, somatomotor, dorsal attention, ventral attention, limbic, frontoparietal, and default mode networks. For held-out collection of 1,000 UKB recordings, we benchmarked various methodologies for classifying parcels into these 7 networks:
1) k-NN classifier using raw parcel time series data. 2) k-NN classifier leveraging parcel embed- dings extracted from a variational autoencoder (VAE). This VAF, equipped with 3 encoding and 3 decoding layers, was trained replicate time series while maintaining the same masking tio and loss metrics as BrainLM. 3) k-NN classifier built on parcel embeddings from a 4-layer Graph Convolutional Network (GCN). This GCN was fine-tuned using a self-supervised masking- centric goal, aiming to master parcel representations with a 75% masking ratio. 4) k-NN classifier grounded on BrainLM's self attentttnn weights between each parcel token and its counterparts. For the classification process, we trained classifiers on 80% of the parcels for each recording, set- ting aside the rest 20% for evaluation. Notably, BrainI.M's attention- driven approach distinctly surpassed the other methods, obtaining parcel classification accuracy of 58.8% (refer to Table 3 In contrast, the k-NN classifier based on the GCN lagged, achieving only 25.9%.
These findings underscore BrainLM's strength in discerning functional brain topography purely from its pretraining phase. The self-attention maps provide meaningful insights about the net- work's identity, despite never encountering explicit labels during the training process.
DISCUSSION
This work presents BrainLM, the first foundation model for functional MRI analysis. By lever- aging self-supervised pretraining on 6,700 hours of brain activity recordings, BrainIM demon- strates versatile capabilities for modeling, predicting, and interpreting human brain dynamics.
A key innovation lies in BrainLM's generalizable representations of fMRI recordings. The model achieves high accuracy reconstructing masked brain activity sequences, even generalizing to held-out distributions. Furthermore, the model improves with an increased number of param- eters showing that our approach scales with data and parameter size. This highlights the ben- efits of large-scale pretraining for learning robust encodings of spatiotemporal neural patterns. BrainLM also provides a powerful framework for biomarker discovery. By fine-tuning, brain dy- namics can be decoded predict clinical variables and psychiatric disorders better than base- line models. This could enable non-invasive assessment of cognitive health using resting-state [MRI alone. Finally, without any network-basse supervision, BrainLM identifies intrinsic func- tional connectivity maps directly from pretraining, clustering parcels into known systems. This demonstrates how self supervieed objectives can extract organizational principles fundamental to the brain.
There remain exciting areas for future work. Multi-modal training could integrate fMRI with ad- ditional recording modalities, such as EEG and MEG, or different brain-wise information such as structural, functional, and genomic data. Probing a wider range of cognitive states and com- bined regularization may yield more generalizable representations. In future work, we could assess zero-shot classification on expanded functional atlases beyond the 7 networks used here. Overall, BrainLM provides springboard for accelerated research at the intersection of neuro- science and AL. We hope that this work spurs further development of foundation models that can help elucidate the intricate workings of the human brain.
9

---

Published as a conference paper at ICLR 2024
6 ACKNOWLEDGMENT
We thank the Yale Wu Tsai Institute for supporting JOC with a Postdoctoral Fellowship, the Yale Graduate Scholars Program for support AHOF and are grateful to Insu Han and Amin Karbasi for their crucial insights and coding assistance in model training. Their contributions have significantly enhanced our research.
REFERENCES
Chadi Abdallah. Brain networks associated with covid-19 risk: Data from 3662 participants. Chronic Stress, 5:2470554022106677777 2021.
Roman Beliy, Guy Gaziv, Assaf Hoogi, Francesca Strappini, Tal Golan, and Michal Irani. From els to pixels and back: Self-supervision in natural-image reconstruction from fmri. Advances in Neural Information Processing Systems, 32, 2019.
Rishi Bommasani, Drew Hudson, Ehsan Adeli, Russ Altman, Simran Arora, Sydney von Arx, Michael S Bernstein, Jeannette Bohg, Antoine Bosselut, Emma Brunskill, et al. On the oppor- tunities and risks foundation models. arXiv preprint arXiv:2108.0725: 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Charlotte Caucheteux, Alexandre Gramfort, and Rean-Rémi King. Deep language algorithms pre- dict semantic comprehension from brain activity. Scientific reports, 12(1):16327, 2022.
Ricky TQ Chen, Yulia Rubanova, Jesse Bettencourt, and David K Duvenaud. Neural ordinary differential equations. Advances in neural information processing systems, 31, 2018.
Zijiao Chen, Jiaxin Qing, Tiange Xiang, Wan Yue, and Juan Helen Zhou. Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22710- 22720, 2023a.
Zijiao Chen, Jiaxin Qing, and Juan Helen Zhou. Cinematic mindscapes: High-quality video re- construction from brain activity. arXiv preprint arXiv:2305.11675 2023b.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre- tre-training of deep bidirectional transformers for language understanding. arXiv preprint 0ttt::1810004003 2018.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010. 1929, 2020.
Andrew Drysdale, Logan Grosenick, Jonathan Downar, Katharine Dunlop, Farrokh Mansouri, Yue Meng, Robert N Fetcho, Benjamin Zebley, Desmond J Oathes, Amit Etkin, al. Resting- state connectivity biomarkers define neurophysiological subtypes of depression. Nature medicine, 23(1):28-38, 2017.
Jennifer Stine Elam, Matthew F Glasser, Michael Harms, Stamatios N Sotiropoulos, Jesper LR Andersson, Gregory C Burgess, Sandra W Curtiss, Robert Oostenveld, Linda Larson-Prion Jan- Mathijs Schoffelen, et al. The human connectome project: a retrospective. Neurolmage, 244: 118543, 2021.
Antonio H de O Fonseca, Emanuele Zappala, Josue Ortega Caro, and David van Dijk. Continuous spatiotemporal transformers. arXiv preprint arXiv:2301. 13338, 2023.
Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked au- toencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Com- puter Vision and Pattern Recognition, pp. 16000-16009, 2022.
10

---

Published as a conference paper at ICLR 2024
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Sebastian Hoefle, Annerose Engel, Rodrigo Basilio, Vinoo Alluri, Petri Toiviainen, Maurício Cagy, and Jorge Moll. Identifying musical pieces from fmni data using encoding and decoding mod- els. Scientific reports, 8(1):2266, 2018.
Tomoyasu Horikawa and Yukiyasu Kamitani. Generic decoding of seen and imagined objects using hierarchical visual features. Nature communications, 8(1):15037, 2017.
Tetsuya lidaka. Resting state functional magnetic resonance imaging and neural network classi- fied autism and control. Cortex, 63:55-67, 2015. ISSN 0010-9452.
NL Johnston- Wilson, CD Sims, JP Hofmann, L Anderson, AD Shore, EF Torrey, and Robert H Yolken. Disease-specific alterations in frontal cortex brain proteins in schizophrenia, bipolar disorder, and major depressive disorder. Molecular psychiatry, 5(2):142-149, 2000.
Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiu:1312.61 2013.
arXiv preprint
Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. arXiv preprint drXiv:160.002900 2016.
Demetres Kostas, Stephane Aroca-Ouellette, and Frank Rudzicz. Bendr: using transformers and contrastive self-supervised learning task to learn from massive amounts of eeg data. Frontiers in Human Neuroscience, pp. 253, 2021.
Te-Jen Lai, Martha E Payne, Christopher E Byrum, David C Steffens, and K Ranga Krishnan. Reduction of orbital frontal cortex volume in geriatric depression. Biological psychiatry, 48 (10):971-975, 2000.
Gengchen Mai, Weiming Huang, Jin Sun, Suhang Song, Deepak Mishra, Ninghao Liu, Song Gao, Taanming Liu, Gao Cong, Yingjie Hu, et al. On the opportunities and challenges of foundation models for geospatial artificial intelligence. arXiv preprint arXiv:2304, 067998, 2023.
Karla Miller, Fidel -llaaoA, Neal K Bangerter, David L Thomas, Essa Yacoub, Junqian Xu, Andreas J Bartsch, Saad Jbabdi, Stamatios N Sotiropoulos, Jesper Andersson, al. Multi- modal population brain imaging in the uk biobank prospective epidemiological study. Nature neuroscience, 19111))5523 523-1536,, 2016.
Milad Mozafari, Leila Reddy, and Rufin VanRullen. Reconstructing natural scenes from fmni pat- terns using bigbigan. In 2020 International joint conference on neural networks (JCNN), pp. 1-8. IEEE, 2020.
Samaneh Nemati, Teddy Akiki, Jeremy Roscoe, Yumeng Ju, Christopher Averill, Samar Fouda, Arpan Dutta, Shane McKie, John H Krystal, JF William Deakin, et al. A unique brain connec- tome fingerprint predates and predicts response to antidepressants. IScience, 23(1), 2020.
Laurel Orr, Karan Goel, and Christopher Ré. Data management opportunities for foundation models. In CIDR, 2022.
Furkan Ozcelik, Bhavin Choksi, Milad Mozafari, Leila Reddy, and Rufin VanRullen. Reconstruc- tion perceived images from fmri patterns and semantic brain exploration using instance- conditioned gans. In 2022 International Joint Conference Neural Networks (IJCNN, pp. 1-8. JEEE, 2022.
Alexandre Pasquiou, Yair Lakretz, John Hale, Bertrand Thirion, and Christophe Pallier. Neu- ral language models are not born equal to fit brain data, but training helps. arXiv preprint arXin:2207.03330, 2022.
Diego A Pizzagalli and Angela Roberts. Prefrontal cortex and depression. Neuropsychophar macology, 47()):22-2246, 2022.
11

---

Published as a conference paper at ICLR 2024
Yulia Rubanova, Ricky TQ Chen, and David K Duvenaud. Latent ordinary differential equations for ireeullrrl-samppled time series. Advances in neural information processing systems, 32, 2019.
Gholamreza Galimi-Khorshiie
Gwenaëlle Douaud, Christian Beckmann, Matthew F Glasser, Ludovica Griffanti, and Stephen M Smith. Automatic denoising of functional mri data: com- bining independent component analysis and hierarchical fusion of classifiers. Neuroimage, 90:449-468, 2014.
Bhedita J Seewoo, Alexander C Joos, and Kirk W Feindel. An analytical workflow for seed-based correlation and independent component analysis in interventional resting-state finri studies. Neuroscience research, 165:26-37, 2021.
Yu Takagi and Shinji Nishimoto. High- resolutioo image reconstruction with latent diffusion mod- els from human brain activity. In Proceedings of the TEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14453-14463, 2023.
Walter Wiggins and Ali S Tejani. the opportunities and risks of foundation models for natural language processing in radiology. Radiology: Artificial Intelligence, 4(4):e220119, 2022.
Joel Ye and Chethan Pandarinath. Representation learning for neural population activity with neural data transformers. arXiv preprint (XXi:21188002100 2021.
Emanuele Zappala, Antonio Henrique de Oliveira Fonseca, Josue Ortega Caro, and David van Dijk. Neural integral equations. arXiv preprint arXiv:2209 151990 2022.
12


Reviews:
[{"review_id": "bX7QyNg3sV", "reviewer": "Reviewer_4kZC", "summary": "In this paper, the authors propose BrainLM, a foundation model designed to analyze fMRI brain activity recordings. BrainLM is trained on 6,700 hours of fMRI data using a Transformer-based architecture. Experimental results show that the proposed BrainLM generalizes well to diverse downstream tasks. Through fine-tuning, it can accurately predict clinical variables such as age and anxiety, while in zero-shot learning, it can identify intrinsic brain networks from fMRI data, offering an approach to understanding spatiotemporal brain dynamics.", "questions": "What is the computational cost of the pretraining?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper is written clearly and is well-structured; the motivation is intuitive and the method is easy to follow.\nThe author collected a large-scale fMRI dataset, which enhances the reliability and generalizability of the pre-trained BrainLM. \nExtensive experiments on diverse downstream tasks like clinical variable prediction and brain state prediction are relatively comprehensive. The additional attention analysis also covers the spatial patterns in the brain.", "weaknesses": "Lack of novelty, the MAE-like pretrain model and method used in this paper are reasonable, but they are known in the field.\n\nLack of comparative baselines: The baselines used for both clinical variable prediction and brain state prediction are somewhat limited. Baselines such as SVM are outdated. The unsupervised MAE-like model mentioned in the related work should also be considered for comparison.\n\nBesides random masking and future masking, more mask strategies should be investigated, like uniform masking and parcel-level (spatial) masking.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cLYLrqH1uC", "reviewer": "Reviewer_NWMt", "summary": "The paper describes a new model for fMRI data trained in a self-supervised manner and then fine-tuned to predict a variety of downstream tasks of interest in neuroscience including brain activity forecasting, clinical outcome prediction and brain network identification. The model is shown to enable better performance than a variety of reported baselines on the above tasks.", "questions": "* Was figure 3 randomly sampled or cherry-picked? \n* The generalization numbers ($R^2=0.464$ within dataset or $.278$ across) need to be contextualized -- how do we know they are good?\n* Why are the 1% and 100% training results of interest? Should we not be surprised that more data improves performance, and if it is surprising why not show a trend over more than just two data sizes? \n* 40% dropout seems quite high -- is this correct, or a typo? \n* Stats on fig 5: how were they done? Hard to imagine BrainLM beats LSTM and LatentODE at later timepoints given the visualized error bars, was it something like BrainLM vs the average of the rest such that the effect of NODE dominated? \n* What motivated the specific network architectures chosen? They vary substantially in number of parameters. \n* What were the other training details (e.g. learning rates and their schedules, dropout and other regularization for the SSL training, gradient clipping if any, etc)? These would be fine in an appendix but should be included for reproducibility.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "The paper operates at the intersection of two areas of intense interest (large-scaled foundation models and neuroscience), and it situates itself well in both domains with a clear setup and motivation, and clear illustration of the modeling paradigm chosen. The results look impressive relative to the baselines chosen.", "weaknesses": "My main concerns lie with the fact that the network is very modestly sized for a foundation model, and that the baselines aren't a priori motivated in a way that would support the contributed model as truly SOTA. \n\nRegarding the former, 13M parameters is smaller than the MLP, LatentODE, and LSTM baselines, and much smaller than what I would consider to be even a small foundation model. For example, HuBERT-Base is a 90M parameter model trained on ~1K hours of speech data (12 layers and 8 heads with larger dimensions than BrainLM). So while BrainLM is a model trained on a nontrivial amount of fMRI data using SSL, it is remarkably small for a foundation model. That would be fine if performance were good relative to a set of baselines chosen based on prior results on fMRI decoding, but I'm not sure if this were the case. At least, the paper does not make it obvious (for example, the NODE, LatentODE, and GCN citations point back to the original papers, none of which have neuroscience applications). Thus, the baselines are also novel, so it's unclear how impactful beating them should be. Considering that the UKB data was released in 2016 and the HCP data published in 2021, I would be surprised if there are no baselines in the literature against which the present contribution can be evaluated. Without this contextualization against past work, neither the performance numbers nor having trained a small SSL model on thousands of hours of fMRI data is sufficiently impactful for ICLR in my judgement. \n\nSome additional comments: \n* In discussion of large-scale SSL for neuroscience, the paper would benefit from also considering Kostas et al. 2021 10.3389/fnhum.2021.653659. \n* More sophisticated methods could be used for interpretability than attention maps (e.g. Chefer et al. CVPR 2021). \n* The legend for Fig. 5 has LatentODE, LSTM, and NODE whereas the text talks about ODENets and non-pretrained BrainLM. Please add the non-pretrained BrainLM results or correct the text. In addition, the text could make a clearer mapping from legend to citations (e.g. \"NODE (Chen et al.), LatentODE (Rubanova rt al.)\"). Finally, I am surprised that there is an unpretrained BrainLM baseline in table 4 but not table 3.\n* Something strange seems to be going on with NODE training -- it performs much worse than other methods and gives a large *negative* $R^2$ value in table 4 (likely indicating severe model mismatch, and giving me concern about whether training failed entirely for this model). \n* Considering the claims regarding \"springboard for accelerated research\" via a foundation model, the paper should make explicit that weights and code will be released.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "48FkcXYSpv", "reviewer": "Reviewer_mZsL", "summary": "The paper presents an transformer architecture based model to (a) reconstruct fMRI sequences and (b) predict clinical variables. As fMRI recordings can be seen as multivariable sequences with rich temporal and cross-variable dependencies, such application has a lot of merit and might give rise to a foundational model for fMRI data analysis.\n\nThe architecture of the model follows other built for similar purposes, and is trained to predict masked-out patches of fMRI BOLD sequences. The model is trained on a large corpus of data from UK BioBank, the largest fMRI data repository currently available.\n\nBrainLM shows performance superior to several benchmarks (but: see a question about those below).", "questions": "Table 1: It appears that allowing the model to see more context (less masking) yields significantly better better performance. (a) Have you tried to train the model with even less masking (MR=0.1) and what were the results? (b) If the trend of \"less masking = better performance\" continues, then what would be the limit of this: at what point using less masking will start to hurt generalizability?\n     \nTable 2: It would help the reader a lot, if, in addition to MSE and R^2 values, you would also report the absolute error in the original units of measurement. For general clinical variables such as age it would help the general public to better understand what kind of error the model makes as measured in years (which is immediately understandable to anyone). For clinicians working on GAD, PHQ, etc scales it would also be immediately impressive to see the performance estimate directly in those units.\n\nPage 5 last paragraph: When you were fine-tuning BrainLM to regress metadata variable, did you predict all of the listed variables (age, neuroticism, PTSD, anxiety) at once as 4-head output, or it was a separate fine-tuning process per variable?\n\nTable 3: What does the \"Raw data\" row indicate? For example for \"Age\" we have \"2.0 +/- 0.22\"? Does this somehow show the variability of age in the dataset (unlikely as 2.0 is too small number for age variability)? Or is this the error that a naive linear model makes? Unclear.\n\nFigure 4: \"CLS\" abbreviation is not explained anywhere in the text.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "The paper is written very clearly, one can follow it easily, the material is presented logically, clearly, and at the same time with sufficient amount of details.\n\nI wholeheartedly support the motivation behind his work. Indeed a LLM-based approach to fMRI sequence analysis makes perfect sense, this methodology should yield great results and help fMRI community to overcome multiple issues with heterogeneity of the data, allowing to pull together large dataset enabling us to train powerful models for neuroscience research and clinical application.\n\nThe way the authors approach the problem is in my opinion technically sound and their solution is exactly what one might expect to see in this context. This gives me a lot of confidence in the soundness of their results and future applicability of the BrainLM model.", "weaknesses": "Table 3: In the section on related work you mention some models that were trained on fMRI to narrowly predict just one clinical variable, and, presumably, those models are state of the art in their respective predictions of clinical variable. However, in Table 3 you seem to compare BrainLM against some other results. My question here I guess whether the comparison presented in the table reflects the state of the art achieved by other models in predicting certain clinical variables, and not just state of the art using a certain method. The fact that an LSTM or an SVM performs poorly is not that interesting to see, what would be interesting to see is how much better or worse performance we get when compared to the state of the art method specifically trained to predict GAD-7 for example. Same for other clinical scores, a methods does not have to predict all of them at once. (If, in fact, the methods you use in this comparison are the current state of the art, please let me know that this is the case and I will withdraw this criticism)\n\nTable 3: Are the \"competitor\" results obtained by you training these methods on the same data, or these are the numbers reported in the respective studies by the authors of those methods?\n\nThe worry I have in those questions above is whether the benchmarks presented are the \"strong\" ones.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vRQKCyxOEo", "reviewer": "Reviewer_xyph", "summary": "The authors train a neural network on fMRI recordings of brain activity: similarly to Large Language Models, the network is trained to predict missing values (random or future) with a self-supervised loss. The paper empirically demonstrated that the learned representations can be used in a variety of downstream tasks, even on a different dataset of fMRI recordings (HCP).", "questions": "I believe the authors make interesting contributions and would be willing to raise my score if my concerns and questions (in the weaknesses section) are addressed.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The performance metrics (R and R squared) reported by the authors are impressive given the variability of brain responses across subjects. This ability of the learned representations to generalize beyond the training set (UKB) to a new dataset with new subjects (HCP) is impressive.", "weaknesses": "A standard protocol for evaluating representations \"downstream\" is to fine-tune a linear predictor or \"probe\" on top of the representations [1, 2]. The fact that the fine-tuned predictor is linear is important: it can measure how well the representations \"cluster\" [3], or linearly correlate, with downstream labels. Yet in this paper, the fine-tuned predictor is a nonlinear neural network with three layers. This raises doubts on whether the representations actually correlate with downstream labels (e.g. age), if they need to undergo many nonlinear transformations before the linear prediction layer. Could the authors comment on this?\n\nThe authors use the terminology of \"foundation model\" and \"language model\" which suggest a novel approach to analyzing fMRI recordings; yet, at its core, their model is a neural network trained with a self-supervised loss (e.g. predict random or future missing values) on fMRI recordings. And there is a rich and existing literature of relevant works that is worth acknowledging. Consider for example [4, 5] as well as their own reference sections. With this context, could the authors explain what contributions they make that are novel with respect to the current literature?\n\n[1] Alain and Bengio, Understanding intermediate layers using linear classifier probes, ICLR workshop 2017.\n\n[2] van den Oord et al. Representation Learning with Contrastive Predictive Coding. Preprint 2019. \n\n[3] Wang et al. Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere. ICML, 2020.\n\n[4] Caucheteux et al. Deep language algorithms predict semantic comprehension from brain activity. Nature scientific reports, 2022. \n\n[5] Pasquiou et al. Neural Language Models are not Born Equal to Fit Brain Data, but Training Helps. ICML, 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)