Self-Supervised Learning of Brain Dynamics from Broad Neuroimaging Data
Authors: Armin W Thomas
Christopher Ré
Russell A. Poldrack
Venue: NeurIPS 2022 Accept (2022)
Keywords: pre-training
self-supervised learning
neuroimaging
mental state decoding
natural language processing
language modelling

Abstract: Self-supervised learning techniques are celebrating immense success in natural language processing (NLP) by enabling models to learn from broad language data at unprecedented scales. Here, we aim to leverage the success of these techniques for mental state decoding, where researchers aim to identify specific mental states (e.g., the experience of anger or joy) from brain activity. To this end, we devise a set of novel self-supervised learning frameworks for neuroimaging data inspired by prominent learning frameworks in NLP. At their core, these frameworks learn the dynamics of brain activity by modeling sequences of activity akin to how sequences of text are modeled in NLP. We evaluate the frameworks by pre-training models on a broad neuroimaging dataset spanning functional Magnetic Resonance Imaging data from 11,980 experimental runs of 1,726 individuals across 34 datasets, and subsequently adapting the pre-trained models to benchmark mental state decoding datasets. The pre-trained models transfer well, generally outperforming baseline models trained from scratch, while models trained in a learning framework based on causal language modeling clearly outperform the others.

Self-Supervised Learning of Brain Dynamies from Broad Neuroimaging Data
Armin W. Thomas Department of Psychology Stanford University athms@atanfoo ord. edu
Christopher Ré
Russell Poldrack
Department of Computer Science
Department of Psychology
Stanford University
Stanford University
chrismre@stanto wwwpstttafordddee..
poldrack@stanffrr ord edu
Abstract
Seff-supervised learning techniques are celebrating immense success in natural language processing (NLP) by enabling models to learn from broad language data at unprecedented scales. Here, we aim to leverage the success of these techniques for mental state decoding, where researchers aim identify specific mental states (e.g., the experience of anger or joy) from brain activity. To this end, we devise a set of novel self supervised learning frameworks for neuroimaging data inspired by prominent learning frameworks in NLP. At their core. these frameworks learn the dynamics of brain activity by modeling sequences of activity akin to how sequences of text are modeled in NLP. We evaluate the frameworks by pre-training models on broad neuroimaging dataset spanning functional Magnetic Resonance Imaging data from 11, 980 experimental runs of 1, 726 individuals across 34 datasets, and subsequently adapting the pre-trained models to benchmark mental state decoding datasets. The pre-trained models transfer well, generally outperforming baseline models trained from scratch, while models trained in a learning framework based causal language modeling clearly outperform the others.
Introduction
In mental state decoding, researchers aim to identify certain mental states (e.g., deciding to accept reject a gamble or the experience of happiness or fear) from brain activity [[] To this end, researchers train predictive models correctly identify (i.e., decode) these mental states from measured brain activity. At first sight, this approach seems straightforward, yet researchers interested in training mental state decoding models faced with the challenge that individual neuroimaging datasets, particularly functional Magnetic Resonance Imaging (fMRI) data, are often of high dimension and low sample size, such that individual samples comprise many hundred thousand dimensions while individual datasets include only a few hundred samples for each of tens to hundreds of individuals. In this setting, where the dimensionality of the data far exceed the number of samples, predictive models are prone to overfitting, which severely limits any generalizable insights that can be gained from training these models about the studied mental states and brain activity.
In spite of the low sample size of individual datasets, neuroimaging research has recently entered a big data era, individual researchers more frequently share their collected datasets publicly [2] B] In addition, several efforts have been made to standardize the data structure 4 and preprocessing [ of neuroimaging data. Together, these developments open up new possibilities for the application of pre-training in neuroimaging at scale, by allowing for the pre-training of models on broad, public neuroimaging data that the resulting models be adapted well the datasets collected by individual researchers, without the need for much new data.
A wealth of empirical evidence has demonstrated that models pre-trained on large neuroimaging
datasets achieve better mental state decoding performances on new data, while also requiring less
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

training time and data, than models trained from scratch (e.g.. 6 91) Yet, much of this work is limited by either pre-training models on large but homogeneous datasets, such as data from many individuals who all perform the same few tasks at the same few acquisition sites [10 7 (jeopardising the generalizability of the resulting models due to potential systematic biases in homogeneous datasets,
[ ) or by requiring prepro-preeoeessee input data (e.g., statistical maps summarizing the measured sequences of brain activity (T)... In addition, researchers often use standard supervised pre-training tasks by tasking models with identifying the specific mental state assigned to each sample of the data [ 7 I0 17]. While this kind of supervised training can be fruitful within individual neuroimaging datasets, it is difficult to extend to many datasets, as neuroimaging researchers generally do not adhere to standardized labeling schemes for mental states ||] when assigning labels to the mental states of their experiments. Due to this lack of standardization, it is often unclear whether two datasets contain the same or distinct mental states (for detailed discussion, see (19)
e.g., specific to the acquisition site or experimental paradigm [1]] 12 13 14
Here, we aim to overcome these limitations by leveraging recent advances in self-supervised learning to pre-train deep learning (DL) models on broad neuroimaging data. In particular, we take inspiration from natural language processing (NLP), where the application of self-supervised learning has led to unprecedented breakthroughs in the adaptive capabilities of pre-trained models (e.g.. and test whether modeling sequences of brain activity akin to sequences of text enables the learning of generalizable representations of brain activity. To enable the application of NLP modeling techniques to neuroimaging data, we represent sequences of brain activity akin to the embedded representation of words in sequences of text in NLP 24! that describes whole-brain activity as the activity of a set of (s.t. each vector value indicates the activity of one of the networks at this point in time (26
20
21
22
2
25 by representing each measurement time point as a vector -indeeeeedn brain networks
Contributions. By representing sequences of brain activity similar to the embedded representation of text in NLP, are able devise novel self-supervised learning frameworks for neuroimaging data that are inspired by prominent learning frameworks in NL.,, namely, sequence-to-sequeen autoencoding (27, causal language modeling (0)] and masked language modeling 21. We pre-train DL models with these newly devised frameworks on large-scale neuroimaging dataset comprising IMRI data from 11, 980 experimental runs of 1, 726 individuals across 34 datasets. To our knowledge, this represents one of the broadest neuroimaging datasets used for pre-training to date, spanning a wealth of experimental conditions and a diverse set of acquisition sites. We evaluate the downstream adaptation performance of the pre-trained models in two benchmark mental state decoding datasets a learning framework based on causal language modeling clearly
and show that models pre-trained outperform the others, while all pre-trained models generally outperform baseline models trained from scratch. To enable others to build on this work, we make our code, training data, and pre-trained models publicly available |
2 Methods overview
2.1 Input data: from brain volumes to brain networks
FMRI data are conventionally represented in four dimensions, such that the measured blood-oxygen- level-dependent (BOLD) signal is indicated in temporal sequences S = (Vi,.., V.} of 3-dimensional volumes V R that show the BOLD signal intensity for each spatial location of the brain (as indicated by the three spatial dimensions x. y. and 2). However, due to the strong spatial correlations of brain activity, the measured BOLD signal is often summarized differently by representing it as set e = (0,,.., Jn of n brain networks (i.e., parcels) 0, each describing the BOLD signal for some subset of voxels Un V with correlated activity patterns.
Here, we utilize such a functional parcellation of brain activity to represent the measured BOLD signal, namely, Dictionaries of Functional Modes (DiFuMo). which was recently proposed by and learned across millions of fMRI volumes. DiFuMo defines θ through a sparse dictionary matrix D E _ containing n p-dimensional networks (where each dimension indicates a voxel e V. flattened over the three spatial dimensions). The BOLD signal of volume V, is then described as linear combination of these networks by the use of weights o E R such that V Do, and ar = D'V, where D indicates the pseudo-inverse of D: DJ = (D" D) D E Raxp, In this
2
github.cm//attms//emnnnng frtm:
2

---

Autoencoding
Causal Seq. Modeling
Sequence-BER1
Network- BERT
time
lime
Seq
seq. B
B60, A
500,
sequence
brain nets extraction
parcellated sequence
Sqp
masii
cls.
sepp
Linear
Linear
Linear
sequence embed.
TR embed.
BI-LSTM
embed.
seg.
encoding
Decodar
LSTM
Transt. Encodor
outputs
Linear Proj.
reconstr: séquence
reconstruct sequance
pradict masked next timepoint
MPP
Task
predict masked
MUP
Task predict masked timeseries
timepoints
Task 2:
Task 2:
Does seq. follow seq, A?
Does seq. B follow seq, A?
Figure 1: Proposed self-supervised learning frameworks for neuroimaging data, inspired by NLP.
work, we utilize the DiFuMo parcellation with = 1024 networks, such that the resulting parcellated BOLD data X E R describe the measured BOLD signal of each of these networks for each time point & of the input sequence S. This representation of the input data is analogous to the embedded representation of sentences NLP, where each word is represented by some embedding vector 24 25
2.2 Self-supervised learning frameworks: modeling sequences of brain activity akin to sequences text
We first establish several commonalities among the frameworks before describing them individually in more detail (for overview, see Fig.
2.2.1 Commonalities among frameworks
First, each framework takes as input a parcellated BOLD sequence X RRxx (with time points for networks) as well as the corresponding sampling frequency (i.e., repetition time or TR) at which the BOLD data were collected (typically in the range of 0.7 to s).
Second, all frameworks linearly project the parcellated BOLD sequence X into an embedding representation x R E be + Σ,, Xt,r Wn.es where w R and b E Re indicate weights and biases, thereby reducing the dimensionality of the input: Rxxx= R
Third, to account for any differences in the sampling frequency of the BOLD sequences, the frame- works add one of learnable TR embeddings E E grxe to each time point & EX, Each TR embedding corresponds to one possible time point & of the input. We set r = 500 with a time increment of 0.2 s between subsequent TR embeddings, such that k O, 0.2, .... 300 s. For example, for an input sequence of 21 that was collected at TR of 0.7 s, we first round each time point t O. 0.7. ..., 21 the closest & and then add the corresponding E7 to E. Note that we always label time points relative to the beginning of an input sequence, such that time point k = 0 is assigned to the first sequence element. Importantly, TR embeddings carry different information than the position embeddings commonly used in transformer models 20 because the same position in an input can correspond to different time points, depending on the sampling frequency at which the data were collected.
3

---

Fourth, all frameworks utilize standard sonnence-tt-eqqueeee DL model architectures f-) that take some embedded sequence E E xx as imput, e.g., defined as the sum of imput and TR embeddings: E = {E + ETRYEE and predict a corresponding output sequence f(E") E R
Fifth, during upstream learning, the output sequence f(E"n) is linearly projected back to the dimen- sionality of the parcellated BOLD sequence X, such that Xt, b,, Ee )t."W... where and b again indicate weights and biases. This step is crucial, as all proposed learning frameworks involve reconstructing X (or parts of X) after some of its elements have been masked it has been encoded in a liverddimnnsionl representation. To measure reconstruction performance, we utilize the mean absolute difference between the parcellated BOLD sequence X and its reconstruction Xt Lree m(J) [x, X₁). where J indicates the set of elements that is to be reconstructed and n(J) the number of elements in J.
Sixth, for downstream adaptation to a new mental state decoding task, the frameworks utilize a simple decoding head p(-). composed of dense hidden layer with model units (one for each embedding dimension, with tanh activation) as well as a so sofar output layer with one model unit i for each considered mental state in the data, p(x);; therefore indicates the probability that input belongs to mental state i. Consequently, all frameworks treat the decoding task as a multinomial classification problem and use standard cross entropy loss objective: Lele Σ. yr log p(z); where y/ indicates a binary variable that equals if is the correct mental state and 0 otherwise.
2.2.2 Autoencoding
The autoencoding framework is inspired by recurrent neural machine translation models (e.g.. These models take as input some text sequence, encode this sequence in a ower-dimensional repre- sentation, and predict an output sequence from the lower-dimensional encoding (c.g.. a translation to another language).
27
Accordingly, the autoencoding framework uses a standard recurrent autoencoder architecture fa(-). with an encoder and decoder part, each comprised of a stack of long short-term memory (LSTM 5CO) units. The encoder e(-) takes E as input and encodes it in lower-dimensional representation h E Re. The decoder d(-) then takes h as input and predicts a corresponding output sequence d(h) Rixe, such that fa(E = d(h). Our encoder is composed of a stack of bidirectional LSTM units, whereas the decoder is build from stack of unidirectional LSTM units. Note that we apply residual connections between the LSTM units in a stack: Let L'S and LST be the i-th and 11th LSTM units in stack (with weights w² and 1)) At the 1-th step of an input sequence, the inputs and outputs of units and (i + 1) are defined as: M4 of = LST M' (h ci i w²); xf = h4 + f+1 c = LSTM 2+1 (ht f1
o .+1 .xf; 1 where x E R represents the input to LSTM' at step t, whereas hū E I and cf E IR represent hidden and cell state respectively. We define the sequence encoding / that is forwarded to the decoder as the mean of the final hidden states of the two LSTM units contained in the encoder's s final bidirectional LSTM unit.
Upstream learning. During pre-training, we jointly train the encoder and decoder to reconstruct the parcellated BOLD sequence X by minimizing its mean absolute difference to the reconstructed se- quence X (see section 2.2.1 min & Et= 1 ΣØ= [&a, - X1,l-- We also apply teacher-forcing [ by using E in 50% of the cases as each next input for the decoder instead of its own preceding output.
Adaptation. To adapt the autoencoding framework new decoding task, we forward input to corresponding decoding head p(-) (see section 2.2.I). thereby omitting the decoder d(-).
2.2.3 Causal sequence modeling (CSM)
The causal sequence modeling (CSM) framework is inspired by recent advances in causal language modeling [20], where models are trained to predict the next word in text sequence. Specifically, these models receive some text sequence as input, represented as matrix X E Ru with n words each encoded by an e-dimensional vector, where the last sequence element (representing the last word) is masked, for example, by replacing it with learned mask embedding Emnkk E R (e.g.. "When it rains, it [msk]"). The model's task is then to predict the masked word. In causal language

---

modeling, models are thereby solely concerned with the part of the input sequence that precedes the masked word.
To adapt causal language modeling to neuroimaging data, we first mask the last time point T of parcellated BOLD sequence X by replacing it with a learnable mask embedding F before transforming X to E by means of linear projection and the addition of TR embeddings ETR. In line with recent advances in causal language modeling, we utilize standard transformer decoder model fa()) (based on the GPT architecture 120) in this learning framework As transformer models are not aware of the ordering of their input, we further add learnable position embeddings E RRx to E each representing one of possible positions in the input sequence (we set p = 512 for all transformer models). Given ESh, the transformer decoder model predicts a corresponding output sequence fa(E E I in the form of the hidden states at the output of its last layer.
Upstream learning. In line with causal language modeling, the upstream objective of the CSM framework is to reconstruct the masked last time point I of the parcellated BOLD sequence X by minimizing the mean absolute difference to its reconstruction Xy: min 1/2 E 1x,, - Xr,al-
Adaptation. During downstream adaptation, we attach a learnable classification embedding
Edl E IR" to the end of BOLD parcellation X and forward the resulting prediction f(E +1 to corresponding decoding head p(-) (see section 2.2.1
2.2.4 Sequence-BERT
The Sequence-BERT framework is based on recent advances in bidirectional masked language modeling, particularly BERT (21] BERT is trained to jointly solve a masked- language-modelinng and pext-sentence-prediiiiee prediction task. To this end, BERT receives two masked sentences S (c.g., "The sky [msk].") and 5h (e.g.. "The [msk] shining.") as input, in which some fraction of words have been randomly masked. BERT is then asked to i) predict the masked words (masked- laaeeeed-lnnggg--mmoode and ii) determine whether 5 follows S" (next-sentence peweeenncceeeeitionn..... In contrast to causal language modeling, masked language modeling is considered as a bidirectional learning task, as the model not just concerned with the parts of the input that precede masked word but also the parts that follow it.
To adapt BERT to neuroimaging data, we first sample two input sequences X" Rxxx= xn and X² E _e xn from the data (with lengths t and tb). In 50% of the cases, the two sequences represent sequential parts of the same underlying BOLD sequence. while in the other 50% of cases, the two sequences are randomly sampled from two distinct BOLD sequences. Due to the strong temporal autocorrelation of BOLD data, we randomly leave a gap of 1-5 TRs (generally corresponding to a gap of to 20 s) between the two sequences when sampling them from the same underlying BOLD sequence (to encourage the models to consider the entire input sequences and not just their connecting time points). In line with BERT, we next randomly mask time points of each sequence (with a probability of 20%) by replacing them with a learnable mask embedding Emak, We also attach a separation emhedding FeeP E R." to the end of each sequence (to indicate their endpoints) and a classification embedding gets to the beginning of S (for the next-sentence- -eenteenc--rediction task; see upstream learning below) E The resulting versions of X" and X are then linearly projected
to obtain respective BOLD embeddings E and x to which we further add TR and position embeddings E and Epos as well as sequence ID embeddings End e 2xe to indicate each of the two input sequences. The resulting embedding representation E = [[Ex + FT ET
Eidgrer" {E,** + ETR + E. + E is then forwarded to standard transformer encoder model f-)- (based on the BERT architecture
(2I)), which outputs a corresponding output sequence fa(Fi") E Ix* in the form of the hidden states at the output of its final layer.
Upstream learning. Similar BERT, we define the upstream objective of the Sequence-BERR framework as the sum of the learning objectives of its two tasks: In line with masked- language-
"While the described input masking procedure is not needed for transformer decoder models. due to their causal attention mechanism, we chose this formulation of the task to ensure that the CSM framework also generalizes to other model architectures.
'We use the same learnable "dummy" TR embedding (see section 2.2.1 and Fig. for any elements that are inserted into an input sequence during training (e.g., for classification or separation embeddings).
5

---

modeling, we define the first objective as the reconstruction loss Lree for all masked time points J:
X,il- In line with next-sentence-preeiicioo we define the other Lree n(J) Σj E Xx
objective as the binary cross entropy loss for a decoding head that receives f(E")) (corresponding to the transformer output for classification embedding Edde) as input and predicts probability p that X follows X Leis -y -ylogg p (1 y) log(l p). where indicates a binary variable that
equals if the two sequences come from the same IMRI run and 0 otherwise.
Adaptation. During downstream adaptation, we forward the transformer's output for classification embedding fe(E"")1 as input to corresponding decoding head p(-) (see section 2.2.1
2.2.5 Network-BERT
All so far presented frameworks model their input on the level of the individual time points of a sequence by trying to reconstruct the parcellated whole-brain BOLD signal for specific time points. Yet, the dynamics of brain actívity data can also be viewed differently by focusing on the interaction of network activities over time instead of the distribution of whole-brain activity individual time points. Consequently, we created a variant of the Sequence-BERT framework that randomly (at a rate of 10%) masks the activity time courses of individual networks n in the parcellated BOLD sequence B
X by replacing them with learnable network mask embedding E E R' (instead of masking whole-brain activity at individual time points). With the exception of the masking procedure, the Network-BERn and Sequence- Dequence-BERT BERT frameworks are identical.
Upstream learning. During upstream learning, Network Betwork-BERT uses the same cross-entropy loss Las as Sequence-BENE and adds it to variant of the reconstruction loss that measures reconstruction performance for the masked network time courses J: I.rec ..j Ejer 7 Ei= Xx. X
Adaptation.
During downstream adaptation, Bewwork-BRR'S forwards the transformer's output for classification embedding fe (F to a respective decoding head p(-) (see section 2.2.1
2.3 Training details
We train all models with stochastic gradient descent and the ADAM optimizer (with 31 0.9, 32 = 0.999, and e = 1e- (32) if not reported otherwise. We also apply a linear learning rate decay schedule (with a warmup phase of 1 1% of the total number of training steps), gradient norm clipping at 1.0, and L2-regularisation (weighted by 0.1).
During upstream learning, set the maximum learning rates to 24-4, 5c 1e-4, and 1e-4 for the autoencoding, CSM, Sequence- Sequence-BERT, and Network- BERT frameworks, respectively (based on the learning rates used in corresponding NLP studies 20 21 (77 ), and randomly sample sequences X of 10 to 55 TRs from our upstream fMRI runs. For the Sequence- and Network-BERT frameworks, each sequence is split into two sequences X and X of equal length (with a randomly sampled gap of 1 to 5 TRs between the two sequences) and X randomly exchanged (with 50% probability) with another X (where + j).
During downstream adaptation, we begin training with the pre-trained model parameters and allow all parameters to be changed freely during training. Input sequences are drawn from the downstream fMRI runs according to the on- and offset defined for each experimental trial (with durations generally ranging between 2 and 30 s), when accounting for the temporal delay of the hemodynamic response function.
Compute resources. Upstream training was performed on Google Compute Engine nl highhemm 64 nodes with four Nvidia Tesla P100 GPUs, 64 CPU threads, and 416 GB RAM memory, while downstream adaptations were performed on compute nodes of the Texas Advanced Computing Center with one Nvidia 1080-TI GPU, 32 CPU threads, and 128 GB RAM memory. Training times varied depending on model size but were generally in the range of 1 to days for upstream learning and 0.5 to 3 hours for downstream adaptations.

---

Experiments
3.1 Datasets
All unprocessed fMRI data used in this study are publicly available through OpenNeuro.org 3 and
the Human Connectome Project (HCP 33 For an overview of our preprocessing, see Appendix
3.1.1 Upstream: 11, 980 IMRI runs from 1, 726 individuals across 34 datasets
Our upstream dataset comprises 11, 980 1MRI runs from 1, 726 in-
dividuals and 34 datasets (see Fig. 2 and Appendix Table ATp. To our knowledge, this represents one of the broadest IMRI datasets used for pre-training in neuroimaging to date, spanning many acqui- sition sites and diverse set of experimental conditions and domains. A few prominent examples of included datasets are The Midnight Scan Club [34.. Individual Brain Charting [351, Amsterdam Open MRI collection [36,, BOLD5000 [77) and the Narratives collec-
34 upstream datscets
20
Runs per Indiiduull
100
80-
80-
25
80
tion We split the upstream data into distinct training and evaluation datasets by randomly designating 5% of the fMRI runs of each included fMRI dataset as evaluation data (at a minimum of 2 runs per dataset) and using the rest of the runs for training. At each evaluation step. we randomly sample 640, 000 sequences from the evaluation dataset.
100
300
Individuels
Figure 2: Number of individ- uals and average number of runs per individual of each up- stream dataset.
3.1.2 Downstream:
benchmark datasets with many mental states
To evaluate the adaptation performance of the pre-trained models, we utilize two benchmark mental state decoding datasets that span wide range of mental states, namely, task-fMRI data from the HCP [3] and the Multi-Domain Task Battery (MDTB 39 ). Our HCP dataset spans 100 participants
and 19 distinct mental states across seven experimental tasks (see Appendix A.2 For each task, two fMRI runs were collected. We also include two resting state IMRI runs per HCP participant, in which individuals simply rest in the IMRI without any particular task, increasing the total number of mental states to 20. The MDTB dataset spans 24 individuals and 26 experimental tasks, which participants performed across two scanning sessions with eight fMRI runs per session. Due the strong similarity of the experimental conditions within individual experimental tasks of the MDTB, we define each of its tasks as one target mental state (see Appendix
3.2 Slyper-paramete evaluation: larger embeddings but not deeper models
In our first experiment, compared the performance of different model hyper-parameter settings in each framework. Specifically, we evaluated three different embedding dimensions e (192, 384, and 768; see section 2.2.D and three different model depths (2, 4, and hidden layers for the encoder and decoder parts of the autoencoding framework and 4, 8, and 12 hidden layers for the transformer models). In line with other work 400 we scaled the number of attention heads n. per layer of the transformer models with the embedding dimension, such that n, 64 We trained one model for each point of the resulting 9-point grid of each framework for 200, 000 training steps, using mini-batch size of 256 sequences. Overall, model performances improved (in terms of validation loss) with larger embedding dimensions but not with model depth (Fig. 3 We therefore decided to continue all further analyses with models comprising 768 embedding dimensions and 2 hidden layers for the encoder and decoder parts of the autoencoding framework and hidden layers for the transformer models.
3.3 Upstream performance: models learn well in all frameworks
In our second experiment, we fully pre-trained one model in each framework according to our previously selected set of hyper-parameters (see section 3.2 by training models for 300, 000 training steps at mini-batch size of 768 sequences. All models learned well over the course of their training,
7

---

Autoencoding
Seq BERT
BERT
0.765
300
1997
13. 199
0.94
15,
19.155
Eval. loss (Lag (e+++)))
0.80
Eval. (kood
0.755
0.74
Evall less
0.750
0.90
0.75
0.745
Evgl loss
0.86
0.70
0.735
0.70
0.730
0.82
100000
200000
0
100000
200000
100000
200000
100000
200000
Training
Training steps
Training
Figure 3: Syper-parameter evaluation. Model performances improve with larger embedding dimen- sions but not with model depth. Note that we do not plot the evaluation loss for the Sequence-BEi variant with 12 hidden layers and 768 embedding dimensions because the loss of this model variant did not meaningfully change over the course of its training (see Appendix B.1
Autoencoding
CSM
Seq BERT
Not-BERT
More +ol
0.9
Loos (Cog Col
0.14
Loss
Loss
0.70
0.60
0.66
10000
150000
800000
10000
150000
300000
10000
180000
300000
10000
150300
800000
Training stops
Training
Training
Training stops
Figure 4: Upstream learning. Models learn well in each framework, with gradually decreasing training and evaluation losses.
Note that the transformer model trained the CSM framework slightly overfitted the upstream data, whereas the gap in training and evaluation performance of the autoencoding framework results from the application of teacher-forcing during training (see section 2.2.2p.
with gradually decreasing training and validation losses (Fig. T..
We also evaluated the brain activity reconstruction performance (as measured by Lyea' see sec- tion 2.2.1 of the final models for different parts of the brain and found that they exhibited similar overall distributions of reconstruction error throughout the brain, with relatively higher errors in the posterior parietal, occipital, and cingulate cortices as well parts of the limbic system (see Appendix B.2).
3.4 Adapting pre-trained language models does not yield upstream performance gains
Given that the formatting of our input data is highly similar to the embedded representation of sentences in language modeling, where each word of sentence is represented by an embedding vector 211,1, we next tested whether adapting pre-trained language models to our upstream data would yield meaningful performance gains, when compared to training models from scratch. To this end, we adapted pre-trained language model variants of GPT-2 4l and BERT [2]] (with 12 hidden layers and an embedding dimension of 768: as provided by Hugging Face's 's Transformer library 42] to our upstream dataset (see Appendix [.3). The upstream validation performance of these two language models did meaningfully improve upon the performance our models trained from scratch (see section 3.3). at final upstream validation losses of 0.73 (CSM with GPT-2) and 0.80 (Sequence-Bil with BERT) (Appendix Fig. B3).
3.5 Downstream adaptation: CSM outperforms other frameworks
Lastly, in our fourth experiment, we evaluated the downstream adaptation performance our pre- trained models in two benchmark mental state decoding datasets (HCP and MDTB; see section B.1.2). To gauge the effectiveness of pre-training, we adapted the pre-trained models to varying sizes of the downstream datasets by randomly assigning individuals to distinct training, validation, and test
8

---

datasets. Specifically, for each evaluated dataset size, we first randomly selected 10 (HCP) and 3 (MDTB) individuals whose data we use for validation, and 20 (HCP) and (MDTB) other individuals whose data we use for testing, before randomly sampling the given number of individuals whose data we use for training (1 to 48 (HCP) and 11 (MDTB)) from the remaining pool of individuals. We then adapted each pre-trained model with varying learning rates and numbers of training steps to the training data (see Appendix B.4) and used the model variant that achieved the highest final mental state decoding accuracy in the validation data for any further analyses (for overview of validation decoding accuracies, see Appendix Fig. [6 linear baseline model (inline with the state-of-the- -ftt----t-t---ttteeee-e--------e in mental state decoding - and variant of our decoding head p(-) that we applied directly to the parcellated BOLD sequences X (see Appendix B.5
[6 and B7]. For comparison, we also trained a
Table 1: HCP test decoding performances. Models are trained on varying training dataset sizes (N = 3, 6, 12, 24, 48 individuals) and tested on a distinct dataset (Nrest 20). Fl-scores are macro-averaged and reported with standard errors. Bold font indicates best metric value. Asterisks indicate meaningfully better predictive accuracy than the respective second-ber best model in McNemar test at a = 0.00083 mmultpple-comparioon corrected: 0.005/6).
Framework
Metris
N
12
N
N 48
Lingar
Acc, F1
24.311.69) 19.1
2..
36.2 31.8
26))..
47.0
51.11.79) 791.2.991.3... 35.6
p(X)
Acc. F1
63.6( 79). 10.0
(4.1(+70)) 36.5
#..01+7791,00 09.4
#.78) 60.8
34.0+-70)) 38.9
50.21+7991 79), 76.5
Antoencoding
Acc. F1
60.2(1. 17.7
60.501.73), 60.6
75.011.68), 67.4
62.0(1.60)
77.0
(1.1 81.1
91.511.
CSM
81.2(+.62)-
86
90.1(+
02.7(++41)-
94
36)+ 92.0
Soq
Flc
39.5(1.17)..0
37.2(1.70).22
62.111.791.38
13.0(1.70)..0
65.6
73.0/1.701.65.
89.21.1.11 (9)...9)..00
Not-BEKT
Acc. F1
36.9(=70))
45.8(+770) 31.4
60.0( 43.5
78.4(4700).64.4
82.8(+.00) 76.5
89.81 48), 85.2
Table 2: MDTB test decoding performances. Conventions as in Table
(Nrest = 9).
Framework
Metris
2
N=
2V
Linear
Aec. Fl
36.111.47
12.2
222.0.2
75.611
11.. 12.9
78.101 39), 77.7
x(X)
Acc.Fl
71.811.35
43).66.. 65.4
78.5:2.4).20
20.0
77.61
101. 76.7
68.011 44), 59
Autoeneoding
Aec.
08,8(1.44)668.
18.4
80. 70.8
834(+.36),
77.8
86.7 83),
CSM
Acc.
77.1
69.9
1-
34). 83.1
38.6(+33)+,.
87
90.01+
89.7
Seq-BERT
Acc.Fl
G3.1
46;
41)
(2)5(6.3)) 72.5
86.31+ 33;, 79.6
NeLB
Acc.Fl F1
71.614
43,
321.
34.6 78.8
87.51 52,
The pre-trained models clearly outperformed the linear baseline model across all training dataset sizes, while their performances also scaled well with the number of individuals in the training data (Tables I and at chance-levels of 5.0% (HCP) and 3.9% (MDTB)). Overall, the transformer decoder model trained in the CSM framework clearly outperformed the other models by achieving the highest test decoding accuracies throughout all evaluated sizes of the training dataset. The other pre-trained models performed on par with the decoding head p(-) in smaller training dataset sizes (S 6 (HCP) and < 3 (MDTB) individuals), while clearly outperforming it in larger training datasets. For the HCP data, for which other reported decoding performances exist in the literature, the pre- pre-trained models performed on par with models trained on much larger datasets (often comprising data of many hundred individuals and 93%.
which generally report test decoding accuracies between 80%
A feature ablation analysis of the models' accurate mental state decoding decisions in both datasets further revealed that these decisions strongly depend on the activity of the occipital and inferior temporal cortex as well as parts of the pre- and postcentral gyrus (see Appendix B.6
We also tested whether the reported downstream model performances are stable over the non- deterministic aspects of their training [5] by replicating our adaptation analysis with different random seeds and splits of the data (see Appendix [B7]]. This replication confirmed our results, yielding final test decoding accuracies that were not meaningfully different from our initial analysis (see Appendix Tables
A6
4 Conclusion
In this work, we propose a set of novel self-supervised learning frameworks for neuroimaging data that are inspired by prominent learning frameworks in NLP. By the use of these frameworks,

---

are able to pre-train DL models on broad, public neuroimaging data, comprising many individuals, experimental domains, and acquisition sites. The pre-trained models generalize well to benchmark mental state decoding datasets and generally outperform baseline models trained from scratch, while models pre-trained in a causal sequence modeling framework clearly outperform the others. We hope that this work will inspire others to explore the benefits and limitations of pre-training in functional neuroimaging at scale, using techniques from self-supervised learning.
This work neglects a key aspect of mental state decoding, namely, the ability to draw inferences about the association between the decoded mental states and input brain activity from the trained models. First empirical evidence indicates that attribution methods from explainable artificial intelligence (XAI [4). research are well-suited to provide insights in the mental state decoding decisions of DL models
Limitations.
Further, this work does not provide any insights into how the proposed self-supervised learning frameworks compare to contrastive learning techniques, which
19 4
have demonstrated great empirical success in computer vision 48 and medical imaging 19
Potential negative social impact. We are currently not aware of any direct negative social impacts that could follow from pre-training DL models public neuroimaging data that are de-identified and collected and shared with human consent in a way approved by institutional review boards.
Acknowledgments. We gratefully acknowledge the support of NIH under No. U54EB020405 (Mobilize), NSF under Nos. 1760950, CCF1763315 (Beyond Sparsity), CCF1563078 (Volume to Velocity), and 1937301 (RTML); ARL under No. W911NF-21-2-022
(Interactive Human-Al Teaming); ONR under No. N000141712266 (Unifying Weak Supervision); ONR N00014-20-1- 2480: Understanding and Applying Non-Euclidean Geometry Machine Learning; RM00142012275 (NEPTUNE); NXP, Xilinx, LETI-CEA, Intel, IBM, Microsoft, NEC, Toshiba, TSMC, ARM, Hitachi, BASF, Accenture, Ericsson, Qualcomm, Analog Devices, Google Cloud, Salesforce, Total, the HAI-GCP Cloud Credits for Research program, the Stanford Data Science Initiative (SDSI), the Texas Advanced Computing Center (TACC) at The University of Texas at Austin, and members the Stanford DAWN project: Facebook, Google, and VMWare. The OpenNeuro data archive is supported by NIH grant R24MH1 17179. The U.S. Government is authorized to reproduce and distribute reprints for Governmental purposes notwithstanding any copyright notation thereon. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views, policies, endorsements, either expressed or implicd, of NIH, ONR, or the U.S. Government.
References
[1] J.-D. Haynes and G. Rees, "Decoding mental states from brain activity in humans," " Nature Reviews Neuroscience, vol. 7, no. 7, pp. 523-534, 2006.
[2] C. Horien, S. Noble, A. S. Greene, K. Lee, D. Barron, S. Gao, D. O'Connor, M. Salehi,
J. Dadashkarimi, X. Shen, et al., "A hitchhiker's guide to working with large, open-source neuroimaging datasets," Nature human behaviour, vol. 5, no. 2, pp. 185-193, 2021.
[3] C. J. Markiewiez, K. J. Gorgolewski, F. Feingold, R. Blair, O. Halchenko, E. Miller, N. Hard- castle, J. Wexler, O. Esteban, M. Goncavles, al., "The openneuro resource for sharing of neuroscience data," Elife, vol. 10, e71774, 2021.
[4] K. J. Gorgolewski, T. Auer, V. D. Calhoun, R. C. Craddock, S. Das, E. P. Duff, G. Flandin, S. S. Ghosh, T. Glatard, Y. O. Halchenko, et al., "The brain imaging data structure, a format for organizing and describing outputs of neuroimaging experiments," Scientific data, vol. 3, no. 1, pp. 1-9, 2016.
[5] O. Esteban, C. J. Markiewicz, R. W. Blair, C. A. Moodie, A. I. Isik, A. Erramuzpe, J. D. Kent, M. Goncalves, F. DuPre, M. Snyder, et al., "fmriprep: robust preprocessing pipeline for functional mri," Nature methods, vol. 16, no. 1, pp. 111-116, 2019.
[6] A. Mensch, J. Mairal, B. Thirion, and G. Varoquaux, "Extracting representations of cognition across neuroimaging studies improves brain decoding," PL.S computational biology, vol. 17, no. 5, p. c1008795, 2021.
10

---

[7] A. W. Thomas, U. Lindenberger, W. Samek, and K.-R. Müller. "Evaluating deep transfer learning for whole-brain cognitive decoding," arxiv preprint arXiv:2111.0152,, 2021.
[8] U. Mahmood, M. M. Rahman, A. Fedorov, Z. Fu, and S. Plis, "Transfer learning of fmri dynamics," arXiv preprint arXiv: 111191111101,...1 2019.
[9] T. He, L. An, P. Chen, J. Chen, J. Feng, D. Bzdok, A. J. Holmes, S. B. Eickhoff, and B. Yeo, "Meta-matee matching as simple framework to translate phenotypic predictive models from big to small data," Nature Neuroscience, vol. 25, pp. 795-804, 2022.
[10] Y. Zhang, L. Tetrel, Thirion, and Bellec, "Functional annotation of human cognitive states using deep graph convolution," Neurolmage, vol. 231, p. 117847, 2021.
[11] A. Kragel, M. Kano, L.. Van Oudenhove, H. G. Ly, P. Dupont, A. Rubio, C. Delon-Martin, B. Bonaz, S. B. Manuck, P. J. Gianaros, el al., "Generalizable representations of pain, cognitíve control, and negative emotion in medial frontal cortex," Nature neuroscience, vol. 21, no. 2, pp. 283-28822999 2018.
[12] T. T. Liu, "Noise contributions to the fmri signal: An overview," Neurolmage, vol. 143, pp. 141- 151, 2016.
[13] D. Sahoo and C. Davatzikos, "Learning robust hicrarchical patterns of human brain across many fmnr studies," Advances in Neural Information Processing Systems, vol. 34, pp. 29034-29048, 2021.
[14] P.-H. C. Chen, J. Chen, Y. Yeshurun, U. Hasson, J. Haxby, and P. J. Ramadge, "A reduced- dimension fmri shared response model," Advances in Neural Information Processing Systems, vol. 28, 2015.
[15] M. Yousefnezhad, A. Selvitella, L. Han, and D. Zhang, "'Supervised hyperalignment for mul- tisubject fmri data alignment," IEEE Transactions Cognitive and Developmental Systems, vol. 13, no. 3, pp. 475 490, 2020.
[16] A. Mensch, J. Mairal, D. Bzdok, B. Thirion, and G. Varoquaux, "Learning neural representations of human cognition across many fmri studies," Advances in neural information processing systems, vol. 30, 2017.
[17] X. Wang, X. Liang, Z. Jiang, B. A. Nguchu, Y. Zhou, Y. Wang, H. Wang, Y. Li, Y. Zhu, F. Wu, al., "Decoding and mapping task states of the human brain via deep learning," Human brain mapping, vol. 41, no. 6, pp. 1505-1519, 2020.
[18] R. A. Poldrack. A. Kittur, D. Kalar, E. Miller, C. Seppa, Gil, D. S. Parker, F. W. Sabb, and R. M. Bilder, "The cognitive atlas: toward knowledge foundation for cognitive neuroscience," Frontiers in neuroinformatics, vol. 5, p. 17, 2011.
[19] A. W. Thomas, C. Ré, and R. A. Poldrack, "Interpreting mental state decoding with deep learning models," " Trends Cognitive Sciences, vol. 26, no. 11, pp. 972-986, 2022.
[20] T. Brown, B. Mann, N. Ryder, M. Subbiah, J. D. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, et al., "Language models are sew-shot learners," Advances neural information processing systems, vol. 33, pp. 1877-1901, 2020.
[21] Devlin, M.-W. Chang, K. Lee, and K. Toutanova, "Bert: Pre-training of deep bidirectional transformers for language understanding," arXiv preprint arXiv: arXiv:/810. 118881000400 2018.
[22] C. Raffel, N. Shazeer, Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. Liu, "Exploring the limits of transfer learning with unified text-to-text transformer," Journal of Machine Learning Research, vol. 21, pp. 1-67, 2020.
[23] R. Bommasani, D. A. Hudson, E. Adeli, R. Altman, S. Arora, S. von Arx, M. S. Bernstein, J. Bohg, A. Bosselut, F. Brunskill, el al., "On the opportunities and risks of foundation models." arXiv preprint ar arXiv:2108.07258 2021.
11

---

[24] J. Pennington, R. Socher, and C. D. Manning. "Glove: Global vectors for word representation." in Proceedings of the 2014 conference on empirical methods in natural language processing (EMNLP), pp. 1532-1543, 2014.
[25] T. Mikolov, K. Chen, G. Corrado, and J. Dean, "Efficient estimation of word representations in vector space," arXiv preprint
2013.
[26] S. B. Eickhoff, B. Yeo, and S. Genon, "Imaging- based parcellations of the human brain," Nature Reviews Neuroscience, vol. 19, no. 11, pp. 672-686, 2018.
[27] Y. Wu, M. Schuster, Z. Chen, Q. V. Le, M. Norouzi, W. Macherey, M. Krikun, Y. Cao, Q. Gao, K. Macherey, el al., "Google's neural machine translation system: Bridging the gap between human and machine translation," arXiv preprint arXiv: 1609. 00199, 2016.
[28] K. Dadi, G. Varoquaux, A. Machlouzarides-Shalle K. J. Gorgolewski, D. Wassermann, B. Thirion, and A. Mensch, "Fine-grain atlases of functional modes for fmni analysis," Neurolm- age, vol. 221, p. 117126, 2020.
[29] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention all you need," Advances in neural information processing systems, vol. 30, 2017.
[30] S. Hochreiter and J. Schmidhuber, "Long short-term memory," Neural computation, vol. 9, no. 8, pp. 1735-1780, 1997.
[31] R. J. Williams and D. Zipser, "A learning algorithm for continually running fully recurrent neural networks," Neural computation, vol. 1, no. 2. pp. 270-280, 1989.
[32] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization," arXiv preprint 1rXiv:1412.6900 2014.
[33] D. C. Van Essen, S. M. Smith, D. M. Barch, T. E. Behrens, E. Yacoub, K. Ugurbil, W.-M. H. Consortium, et al., "The wu-minn human connectome project: overview," Neuroimage, vol. 80, pp. 62-79, 2013.
[34] E. M. Gordon, T. O.O.Laan Laumann, A. W. Gilmore, J. J.Weboldd D. Greene, J. Berg, M. Ortega, C. Hoyt-Drazen, C. Gratton, H. Sun, et al., "Precision functional mapping of individual human brains," Neuron, vol. 95, no. 4, pp. 791-807, 2017.
[35] A. Pinho, A. Amadon, T. Ruest, M. Fabre, Dohmatob, Denghien, C. Ginisty, Becuwe- Desmidt, S. Roger, L. Laurier, et al., "Individual brain charting, a high-resolution fmri dataset for cognitive mapping." " Scientific data, vol. no. 1, pp. 1-15, 2018
[36] L. Snoek, M. M. van der Miesen, T. Beemsterboer, A. van der Leij, A. Eigenhuis, and H. Steven Scholte, "The amsterdam open mri collection, set of multimodal mri datasets for individual difference analyses," Scientific data, vol. 8, no. 1, pp. 1-23, 2021.
[37] N. Chang. J. A. Pyles, A. Marcus, A. Gupta, J. Tarr, and E. M. Aminoff, "Bold5000, a public fmri dataset while viewing 5000 visual images," " Scientific data, vol. 6, no. I, pp. 1-18, 2019.
[38] S. A. Nastase, Y.-F. Liu, H. Hillman, A. Zadbood, I.. Hasenfratz, N. Keshavarzian, J. Chen, C. J. Honey, Y. Yeshurun, M. Regev, et al., "TTh "narratives" fmri dataset for evaluating models of naturalistic language comprehension," Scientific data, vol. 8, no. 1, pp. 1 1-22, 2021.
[39] M. King, C. R. Hernandez- Casttillo, R. A. Poldrack, R. B. Ivry, and J. Diedrichsen, "Func- tional boundaries in the human cerebellum revealed by a multi-domain task battery," Nature neuroscience, vol. 22, no. 8, pp. 137 1377-, 2019.
[40] I. Ture, M.-W. Chang, K. Lee, and K. Toutanova, "Well-read students learn better: On the importance of pre-training compact models," arXiv preprint /wt.ivv:/000000 2019.
[41] A. Radford, Wu, R. Child, D. Luan, Amodei, I. Sutskever, el al., "Language models are unsupervised multitask learners," OpenAl blog, vol. 1, no. 8, p. 9, 2019.
12

---

[42] T. Wolf, L. Debut, V. Sanh, J. Chaumond, C. Delangue, A. Moi, P. Cistac, T. Rault, R. Louf, M. Funtowicz, et al., 'Transformers: State-of-the-art art natural language processing," in Pro- ceedings of the 2020 conference on empirical methods in natural language processing: system demonstrations, pp. 38 45, 2020.
[43] M.-A. Schulz, B. Yeo, J. T. Vogelstein, J. Mourao- J. N. Kather, K. Kording, B. Richards, and D. Bzdok, "Different scaling of linear models and deep learning in uk- biobank brain images versus machine-learning datasets," Nature communications, vol. 11, no. 1,
pp. 1-15, 2020.
[44] Y. Zhang, N. et Farrugia, and P. Bellec, "Deep learning models of cognitive processes constriined by human brain connectomes," bioRxiv, 2021.
[45] X. Bouthillier, P. Delaunay, M. Bronzi, A. Trofimov, B. Nichyporuk, J. Szeto, N. Moham- madi Sepahvand, Raff, K. Madan, V. Voleti, et al., "Accounting for variance in machine learning benchmarks," Proceedings of Machine Learning and Systems, vol. 3, pp. 747-769, 2021.
[46] W. Samek, G. Montavon, A. Vedaldi, L. K. Hansen, and K.-R. Müller, Explainable Al: inter- preting, explaining and visualizing deep learning. Springer Nature, 2019.
[47] A. W. Thomas, C. Ré, and R. A. Poldrack, "Comparing interpretation methods in mental state decoding analyses with deep learning models," arXiv preprint arXiv:2205./558/ 2022.
[48] T. Chen, S. Kornblith, M. Norouzi, and G. Hinton, "A simple framework for contrastive learning of visual representations," in International conference on machine learning, pp. 1597-1607, PMLR, 2020.
[49] K. Chaitanya, E. Erdil, N. Karani, and E. Konukoglu, "Contrastive learning of global and local features for medical image segmentation with limited annotations," Advances in Neural Information Processing Systems, vol. 33, pp. 12546-12558, 2020.
[50] G. Xue and R. A. Poldrack, "The neural substrates of visual perceptual learning of words: implications for the visual word form area hypothesis." " Journal of cognitive neuroscience, vol. 19, no. 10, pp. 1643-1655, 2007.
[51] R. A. Poldrack, Congdon, W. Triplett, K. Gorgolewski, K. Karlsgodt, Mumford, Sabb, N. Freimer, E. London, T. Cannon, et al., "A phenome-wide examination of neural and cognitive function," Scientific data, vol. 3, no. 1, pp. 1-12, 2016
[52] M. Hanke, F. J. Baumgartner, P. lbe, F. R. Kaule, S. Pollmann, O. Speck, W. Zinke, and J. Stadler, "A high-resolution 7-tesla fmri dataset from complex natural stimulation with an audio movie." Scientific duta, vol. 1, no. 1, pp. 1-18, 2014.
[53] C.-W. Woo, M. Roy, J. T. Buhle, and T. D. Wager, "Distinet brain systems mediate the effects of nociceptive input and self-regulation pain," PLoS biology, vol. 13, no. 1, e1002036, 2015.
[54] A. Smeets, M. Kroese, C. Evers, and T. de Ridder, "Allured or alarmed: counteractive control responses to food temptations in the brain," Behavioural bruin research, vol. 248, pp. 41 41145, 2013.
[55] J. Koster-Hale, R. Saxe, J. Dungan, and L. L. Young, "Decoding moral judgments from neural representations of intentions," Proceedings of the National Academy of Sciences, vol. 110, no. 14, pp. 5648-5653, 2013.
[56] Chen, Y. C. Leong, C. J. Honey, C. H. Yong, K. A. Norman, and U. Hasson, "Shared memories reveal shared structure in neural activity across individuals," Nature neuroscience, vol. 20, no. 1, pp. 115-125, 2017.
[57] J. Chen, C. Honey, F. Simony, M. J. Arcaro, K. A. Norman, and U. Hasson, "Accessing real-life episodic information from minutes versus hours earlier modulates hippocampal and high-order cortical dynamics," Cerebral cortex, vol. 26, no. 8, pp. 3428-34444 2016.
13

---

[58] I. Momennejad, A. R. Otto, N. D. Daw, and K. A. Norman, "Offline replay supports planning in human reinforcement learning," elife, vol. 7, p. e32548, 2018.
[59] A. Shenhav, M. A. Straccia, S. Musslick, D. Cohen, and M. M. Botvinick, "Dissociable neural mechanisms track evidence accumulation for selection of attention versus action," Nature communications, vol. 9, no. 1, pp. 1-10, 2018.
[60] R. Botvinik-Nezer, R. Iwanir, Holzmeister, J. Huber, M. Johannesson, M. Kirchler, A. Dreber, C. F. Camerer, R. A. Poldrack, and T. Schonberg, "fmri data of mixed gambles from the neuroimaging analysis replication and prediction study," Scientific data, vol. 6, no. 1, pp. 1-9, 2019.
[61] M. Piva, K. Velnoskey, R. Jia, A. Nair, I. Levy, and W. Chang, "The dorsomedial pre- frontal cortex computes task- lusk-invariant relative subjective value for self and other," Elife, vol. 8, p. e44939, 2019.
[62] A. Shenhav, M. A. Straccia, D. Cohen, and M. Botvinick, "Anterior cingulate engagement in foraging context reflects choice difficulty, not foraging value," Nature neuroscience, vol. 17, no. 9, pp. 1249-1254, 2014.
[63] T. Nakai and S. Nishimoto, "Quantitative models reveal the organization of diverse cognitive functions in the brain," Nature communications, vol. 11, pp. 1-12, 2020.
[64] S. Schwettmann, J. B. Tenenbaum, and N. Kanwisher, "Invariant representations of mass in the human brain," Elife, vol. 8, p. e46619, 2019.
[65] J. A. Avery, A. G. Liu, J. E. Ingeholm, C. D. Riddell, S. J. Gotts, and A. Martin, "Taste quality representation in the human brain," Journal of Neuroscience, vol. 40, no. 5, pp. 1052, 2020.
[66] M. E. Sachs, A. Habibi, A. Damasio, and J. T. Kaplan, "Dynamic intersubject neural synchro- nization reflects affective responses to sad music," Neurolmage, vol. 218, p. 116512, 2020.
[67] N. Saadon-Grosman,, S. Arzy, and Y. Loewenstein, "Hierarchical cortical gradients in somatosen- sory processing," Neuroimage, vol. 222, p. 117257, 2020.
[68] R. Soreq, I. R. Violante, R. E. Daws, and A. Hampshire, "Neuroimaging evidence for network sampling theory of individual differences in human intelligence test performance," Nature communications, vol. 12, no. 1. pp. 1-13, 2021.
[69] L. Tomova, K. L. Wang. T. Thompson, G. A. Matthews, A. Takahashi, M. Tye, and Saxe, "Acute social isolation evokes midbrain craving responses similar to hunger," Nature Neuro- science, vol. 23, no. 12, pp. 1597-1605, 2020.
[70] J. W. Antony, T. H. Hartshorne, K. Pomeroy, T. M. Gureckis, U. Hasson, S. D. McDougle, and K. A. Norman, "Behavioral, physiological, and neural signatures of surprise during naturalistic sports viewing." Neuron, vol. 109, no. 2, pp. 377-390, 2021.
[71] J. A. Avery, A. G. Liu, J. E. Ingeholm. S. J. Gotts, and A. Martin, "Viewing images of foods evokes taste quality-specific activity in gustatory insular cortex," Proceedings of the National Academy of Sciences, vol. 118, no. 2, p. e20109321 18, 2021.
[72] E. Knights, C. Mansfield, D. Tonin, J. Saada, F. W. Smith, and S. Rossit, "Hand-selective visual regions represent how to grasp 3d tools: brain decoding during real actions," Journal of Neuroscience, vol. 41, no. 24, pp. 5263-5273, 2021.
[73] L. J. Chang. E. Jolly, J. H. Cheong, K. M. Rapuano, N. Greenstein, P.-H. A. Chen, and J. R. Manning, "Endogenous variation in ventromedial prefrontal cortex state dynamics during naturalistic viewing reflects affective experience," Science Advances, vol. 7, no. 17, p. eab 2021.
[74] B. Fischl, "Freesurfer," Neuroimage, vol. 62, no. 2, pp. 774-781, 2012. 2012.
14

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes] see section 3.3 and 3.5
(b) Did you describe the limitations of your work? (Yes) see section
(c) Did you discuss any potential negative societal impacts of your work? [Yes] see section
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or URL)? [Yes] see section
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] see sections 2.3
3.5 and Appendix B.5 and B.4
(c) Did you report error bars (e-g.. with respect the random seed after running experi- ments multiple times)? [N/A] We replicated our downstream adaptation analysis with different random seeds and splits of the data and found no meaningful differences in final test decoding accuracies of the individual training runs between the initial analysis and replication (see Appendix B.7)
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] section 2.3
4. If you using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes] see section 3.1 and
Appendix [.
(b) Did you mention the license of the assets? [Yes] see Appendix A.1
(c) Did you include any new assets either the supplemental material or as URL? Yes] we make our pre-trained models and training data publicly available; see section)
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] see section and Appendix A.
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [Yes] see section and Appendix [..
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
15


Reviews:
[{"review_id": "edkqbYfZ6nO", "reviewer": "Reviewer_JeMm", "summary": "In this work, the authors benchmark a number of NLP architectures in the task of learning representations from fMRI data. The architectures include a seq2seq, transformer, and transformer-based BERT models. Unlike the original NLP models trained with cross entropy loss, the reconstruction loss is used in the presend work instead. The proposed objectives were pre-trained on 34 and evaluated on two publicly available datasets. The results show applicability of self-supervised pre-training for downstream tasks such as mental state decoding. Potentially interesting work and research direction but with a weak evaluation and inconclusive utility of the approach.\n", "questions": "- Would the results and ranking still hold if the embeddings are properly evaluated using metrics not sensitive to a possible data imbalance (see details above)?\n- How would the results compare with supervised embeddings (see details above)?\n- How does $p(\\cdot)$, the same model as used now for everything but the baseline, applied to the input $X$ rank against other models (see details above)?", "limitations": "- An inconclusive demonstration of utility of the proposed approach (major).\n- Focus on the pre-training approaches based on masking and lack of mention or comparison with contrastive self-supervised learning, which shows strong results in a similar context.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "## Strengths\n- A novel application of causal and masked data modeling to fMRI data.\n- A demonstration of viability of the masked self-supervised training for pre-training of models embedding fMRI, as shown in the transfer learning on a state-decoding task on fMRI. However, the experiments are limited to one downstream task on two datasets. Evaluation and the training is also problematic (see the Weaknesses below)\n- A wide variety of considered NLP approaches.\n\n## Weaknesses (approximately in the order of importance)\n- Unfair evaluation. Unlike the cross entropy loss in the original NLP applications of the employed models, the reconstruction error introduces a problem in all models but the seq2seq autoencoder, where the bottleneck representation $h$ is used for downstream task. The downstream evaluating model $p(\\cdot)$ needs to be a part of the evaluation on the input X, not just the \"Linear Baseline\" from the paper. Main concern is that the learned embeddings and the input data X are only a single linear transform apart from each other. If not that transform, the CSM and BERT models would be encouraged to learn identity functions, up to a linear autoencoder.\n- Inconclusive evaluation. Only accuracy numbers are shown, but the numbers highly dependent on the balance in the dataset. To avoid gross misinterpretations the field uses now standard approaches such as ROC AUC or F1, which in the multiclass case will OVO MACRO or similar. Balanced accuracy is also a better choice. A confusion matrix would reveal some details, but still require a sumarization.\n- Evaluation on the downstream task should not simply refer to Appendix but present confidence intervals and statistical significance of improvements. The standard way of comparing performance. Since this is an application paper the rigor of comparison is the most important metric of paper quality. For example, loss curves are unimportant as uninformative, since we're evaluating the utility of considered self-supervised approaches and a closer focus on downstream task, including the model variability.\n- Lacking comparison with models pre-trained in supervised manner. Given the data, all of the models (architectures) considered in the manuscript could have been pre-trained with the downstream model attached, i.e. $p(f(X))$ trained to predict the states, but only the obtained embedding used further in evaluation. It may happen that supervised embedding would be as or more efficient than the proposed approach.\n- The experiments do not support the claim that training on the homogenous dataset limits the generalizability of the pre-trained models. Moreover, the self-supervised or supervised learning objectives are agnostic to the homogeneity of the dataset. Thus this is a limitation of the experimental setup in the related work rather than the limitations of the pre-training models. Combining multiple inhomogeneous datasets in the medical imaging domain might lead to underspecification and shortcuts that the model can use. Hence, researchers need to control for additional factors (e.g., site effect, age, gender). To support the claim, further experiments are needed to see how the number of datasets and their homogeneity affects the generalizability of self-supervised learning objectives and whether it is a limiting factor.\n- The authors claim that the data is minimally-preprocessed. However, this and related work follows = standard preprocessing such as spatial smoothing and detrending. Common choices for preprocessing pipelines are fMRIPrep, C-PAC, or SPM. In addition, the authors use DiFuMo, which is a matrix factorization method, which itself is a way to embed fMRI data. The authors need to clarify what is minimal. Moreover, self-supervised objectives would be agnostic again.\n- Note, the statements about the way BERT is trained are not technically correct. BERT is pre-trained either on Masked LM or Next Sentence Prediction. Not Next Sentence Prediction and reconstruction error simultaneously as used in the manuscript.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Yo3yEDoXCv", "reviewer": "Reviewer_ca3n", "summary": "The authors have extensively experimented with adapting some well-known NLP models to neuroimaging datasets. They have successfully trained autoencoding, causal, and BERT-styled models on either predicting the entire input sequence or predicting parts of the input sequence (mask)\nThey achieve excellent results in the downstream tasks, especially with the causal seq models, and have shown the generalizability of the trained model to achieve good results on the as-yet unseen datasets.\nThese types of research are important to further the field of machine learning in neuroimaging as labels are really hard to come by.", "questions": "1. Can you explain why you needed a preliminary dimensionality reduction using DiFuMo? And instead just start with X in R^(txp) space where p is the number of voxels (at least in the grey matter)?\n\n2. Why did you restrict yourself to brain decoding and not mental disorders?\n3. Why isn't the training code being made available?", "limitations": "Brain dynamics often is defined by the interaction between networks. It would be have been great if there were some ablation studies done to check which interaction or single regions were most informative to the downstream task.\n", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nA large and extensive dataset was used.\nNumber of key models from NLP has been adapted.\nThe pretext task is straight-forward and simple.\n\nWeaknesses:\nThe pretext tasks don't have anything particular to tie them to predicting brain states. The authors could have used many a datasets out there to predict something useful as in autism or depression as a downstream task.\nWhy did causal models have to perform better than the rest? No investigation on this has been attempted.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "XmLXIvNgooH", "reviewer": "Reviewer_R4Ef", "summary": "In this paper, different self-supervised learning architectures are adopted from NLP literature to predict brain activity. Specifically,\n\n1. Autoencoders using LSTMS\n2. GPT-style autoregressive models\n3. BERT-style masked autoregressive model and a next sentence prediction task adapted to fMRI data (are the two sequences from the same dataset or not)\n4. Same as 3 but a particular network (akin to an emebdding dimension) is dropped from all input embeddings\n\nThe paper explores training all 4 architectures using data from a large collection of fMRI experiments, and finds that all models train well. While there is no clear effect of model depth, models with higher capacity perform better.\n\nFinally, the pretrained models are evaluated on two downstream datasets (for task classification) and are all found to beat traditional linear baselines.", "questions": "Questions:\n1. (Lines 237-238) Isn’t 1-5 TRs too small a gap considering HRF? Have the authors experimented with longer gaps?\n\n1. Does the model use low-level cues like different repetition times for the task equivalent to next-sentence-prediction? One way to analyze this would be to correlate CE with some measure of difference in the TRs. Similarly, what about cues like participant ID or task type given that specific networks will be active depending on the task?\n2. I am not sure I follow the experiment in section 3.4. If the experiment is to use GPT/BERT pretrained on text and then finetuned on the fMRI setup, why would that work?\n3. How is the data quality/SNR related to training/validation/test loss? This was not clear to me but did the original pretraining also split train/val/test such that no participant contributed data to more than one set?\n\nMisc. questions unrelated to my views of the current manuscript:\n1. What is the relationship of ablating differentiating diff. networks on the downstream task accuracy? Is there a possibility for meaningful relationships between the same?\n2. Relatedly, what is the impact of ablating a specific network on the MAD of different networks in the output? Is there some functional, anatomical organization to the same?", "limitations": "-NA-", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Edit after rebuttal: I believe that the authors have sufficiently answered my questions and thought the response on using pertained GPT representation was especially interesting. I agree with other reviewers that in its current form, the usability of the networks for real-world downstream tasks is limited. However, I believe this paper is, to the best of my knowledge, highly novel and will spur interesting further research. I recommend it for acceptance.\n\n#########################################################################################################\n\nStrengths:\n1. Clarity: This paper was very well written, easy to understand and well visualized. I enjoyed reading it!\n2. Quality: I found the results presented convincing, although there is a lot to unpack and it is not fully apparent to me how the choice of hyper parameters, data construction schemes etc. affect the observed results\n3. Originality & Significance: To the best of my knowledge, this is the first study to explore such pretraining schemes for fMRI data. It is a very exciting and significant direction for future research!\n\nWeaknesses:\n1. It took me a while to understand the need for both positional encodings and the TR encodings. Clarifying this earlier would improve readability in my opinion.\n2. I would urge the authors to add standard error to the values in fig.5 over many random seeds.\n3. Several (all?) of the analyses do not currently report standard error. Given the number of design choices made throughout, I believe this would be an important addition to improve the paper.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept