SIM: Surface-based fMRI Analysis for Inter-Subject Multimodal Decoding from Movie-Watching Experiments
Authors: Simon Dahan
Gabriel Bénédict
Logan Zane John Williams
Yourong Guo
Daniel Rueckert
Robert Leech
Emma Claire Robinson
Venue: ICLR 2025 Poster (2025)
Keywords: movie-watching experiment
fMRI
cortical analysis
surface-based transformers
multimodal learning
contrastive learning
self-supervised learning
generalization
encoding/decoding
Primary area: applications to neuroscience & cognitive science
TL;DR: A surface-based deep learning fMRI model that generalises encoding and decoding of audio-visual stimuli from movie-watching experiments to unseen subjects and unseen stimuli

Abstract: Current AI frameworks for brain decoding and encoding, typically train and test models within the same datasets. This limits their utility for cognitive training (neurofeedback) for which it would be useful to pool experiences across individuals to better simulate stimuli not sampled during training. A key obstacle to model generalisation is the degree of variability of inter-subject cortical organisation, which makes it difficult to align or compare cortical signals across participants. In this paper we address this through use of surface vision transformers, which build a generalisable model of cortical functional dynamics, through encoding the topography of cortical networks and their interactions as a moving image across a surface. This is then combined with tri-modal self-supervised contrastive (CLIP) alignment of audio, video, and fMRI modalities to enable the retrieval of visual and auditory stimuli from patterns of cortical activity (and vice-versa).  We validate our approach on 7T task-fMRI data from 174 healthy participants engaged in the movie-watching experiment from the Human Connectome Project (HCP). Results show that it is possible to detect which movie clips an individual is watching purely from their brain activity, even for individuals and movies *not seen during training*. Further analysis of attention maps reveals that our model captures individual patterns of brain activity that reflect semantic and visual systems. This opens the door to future personalised simulations of brain function. Code \& pre-trained models will be made available at https://github.com/metrics-lab/sim.

Published as a conference paper at ICLR 2025
SIM: SURFACE-BASED FMRI ANALYSIS FOR INTER- SUBJECT MULTIMODAL DECODING FROM MOVIE- WATCHING EXPERIMENTS
Simon Dahan' 1+ Gabriel Bénédict 1 Logan Z. J. Williams'
Yourong Guo'
Daniel Rueckert3.45 Robert Leech Emma C. Robinson'
1 RRseaarch Department of Biomedical Computing, BMEIS, King's College London
2 "Amazon, Madrid
3 Innttitute for AI in Medicine, Technical University of Munich
*Department of Computing, Imperial College London
Munich Center for Machine Learning (MCML)
Innttitute of Psychiatry, Psychology & Neuroscience, King's College London
simon. dahankkcl..c..... ac. uk, emma robiisonkkkl....ccccoo.. acc.k
ABSTRACT
Current AI frameworks for brain decoding and encoding, typically train and test models within the same datasets. This limits their utility for cognitive training (neurofeedback) for which would be useful to pool experiences across individ- uals to better simulate stimuli not sampled during training. A key obstacle to inter-subject cortical organi- model generalisation the degree variability cortical signals across partic- sation, which makes it difficult to align or compare ipants. this paper we address this through use of surface vision transformers, which build generalisable model of cortical functional dynamics, through en- coding the topography of cortical networks and their interactions as a moving image across surface. This is then combined with tri-modal self-supervised contrastive (CLIP) alignment of audio, video, and fMRI modalities to enable the retrieval of visual and auditory stimuli from patterns of cortical activity (and vice-versa). We validate our approach on 7T MMR data from 174 healthy participants engaged the movie-watching experiment from the Human Con- nectome Project (HCP). Results show that it is possible to detect which movie clips an individual is watching purely from their brain activity, even for individ- uals and movies not seen during training. Further analysis of attention maps reveals that our model captures individual patterns of brain activity that reflect semantic and visual systems. This opens the door to future personalised simu- lations of brain function. Code & tre-trained models will be made available at pttpps//qitthnn
INTRODUCTION
Over recent years, there has been growing interest in the extent which machine learning frame- works, such as convolutional neural networks (CNNs) and transformers, can model neurological
processes: from spatial encoding of the hippocampus (Ellwood 2024 Whittington et 202T Kim
et al.
2024) to replicating semantic (Antonello & Huth 2024; Caucheteux et al., 2023 Huth et al.
2016
Millet et 2022) and visual (Benchetrit et al. 2023] Ozcelikk VanRullen 2023 Tang al.,
2024
Wen et al.) 2018) representations within the cortex. These approaches support the testing of
new
theories of human cognition (Millet et al. 2022 Antonello & Huth 20224] Caucheteux et al.
2023
Ellwood 2024
Kriegeskorte 2015; Whittington et al.] [202] Kim et al. 2024). and allow
for the encoding or decoding of stimuli [Benchetrtt et al. [ Défossez et al. [003]] Gu et al.
2022
Lindsay 2021
Kriegeskorte
2015
Ozcelik & VanRullen 2023 Scotti et al. 2024 Thomas
Equual contribution
Work done outside of Amazon

---

Published as a conference paper at ICLR 2025
Input Sphere
Native Mesh
MMRI
B
A
Potthh Partitionn
No 1780VV 45
V 40962
V- 59292
4
D
tri-modal CLIP Loss
MdAI Marpper
fi Vi
4 Us
f
My
Mapree
VsMAE
Encodar
Ehooder
Je
fn
0g
audio clip
Video Mapper
f
pre-train model (fozee)
Videoo Enpocies
finetuning
video clip
Figure 1: (SIM) seeks to align visual and audio stimuli extracted from 7 = second movie clips- with brain activations acquired over the time period: [A] 7T fMRT data, collected during movie watching, is first projected to subjects' native cortical surfaces (resolution V=59292 vertices); then inflated to a sphere, and downsampled onto regularly tessellated icosphere (Lg with V=40962 ver- tices). [B] This is then encoded using surface vision transformer (SiT), which tokenizes data by patching each Ig sphere with a lower-resolution icospheric grid to generate sequence of triangular patches (V=45 vertices per patch; N=1280 number of patches). [C] The SiT encoder (#/M is pre-trained as part of a video surface masked autoencoder (vsMAE) for fMRI-frame reconstruction; [D] and fMRI embeddings (fi) are then aligned with CLIP contrastive training to video (v:) [E] and audio (a,) [F] embeddings learnt from videoMAF. (& mne) and wav2vec ($A) model; and pro- jected to vector spaces of common lenght by multimodal mappers (f/ f .fv foA). At test time this makes it possible decode video/audio stimuli from IMRI (or vice versa) through comparing the similarity of their CLIP embeddings (ft, a,, U.). This comparison generates probability dis- tribution over the candidate samples, allowing for evaluation of retrieval performance using top-K accuracy metrics.
al. 2022 Thual et al. 2023: Wen et al. 2018 Yamins & DiCarlo 2016) from non-invasive human brain recordings such as functional magnetic resonance imaging (fMRI), maggett--nccephalograhh (MEG) or peeetr-eencephalographe typically require large amounts of imaging data per subject, and do not generalise to unseen indi- viduals. This limits the extent to which they can be used to probe individual sources of variation, to ultimately build precision models of human cognition and behaviour.
(EEG). However, such comparative models are subject-speciff
One contributing factor to the lack of generalisation of current models, is arguably that most models treat signals from adjacent locations in the brain spatially independent encoding patterns of brain activity using linear, or non-linear regression modules (Huth et al., 2016 Millet et al.) 2022 Caucheteux et al. 202 Scotti al. 2024 Ozcelik & VanRullen, 2023 This ignores the well- documented spatial auto-correlation of signals across the cortex that is known to be behaviourally
meaningful (Bijsterbosch et al.] 2018 Kong al.] 2019) [Shinn et al. 2023 Leech al. 2023
Margulies et al. 2016).
At a high level, cortical activity may be decomposed into signals from a relatively small number of functionally specialised areas (Glasser et al.] (20 Smith et a (013)). These regions communicate through neuronal connections, resulting in coordinated patterns (or networks) activity. Previous studies have shown that human brain function may be modelled from temporal sequences of these networks (Vidaurre et al.) 2017 Pervaiz et al. 2022 Smith et al. 2013). This suggests that we build an encoding model that generalises these spatial patterns across individuals, then predict what any individual's cortical functional dynamics should be response to given stimulus; predict novel stimulus from the spatial topography of their brain activity.
2

---

Published as a conference paper at ICLR 2025
With this in mind, recent efforts to generalise vision transformers (ViTs) (Dosovitskiy et al. 2020 to the cortical surface present significant opportunity as they allow for the modelling long range spatio-temporal interactions through the encoding of space- self-attention (Dahan et al.) 2022 2024). Relative to a 3D network of equivalent resolution, surface models are much more compact and require less memory; this creates scope for building more sophisticated network architectures. Moreover, surface modelling more faithfully represents the true geometry of the convoluted cortical surface, and for this reason has long been favoured for analysis of cortical signals (Glasser et al. 2016b Margulies et al. 2016 Gordon et al. 2017b a..
Contributions: In this paper, we introduce SIM (Fig D. a novel framework that combines self- supervised contrastive learning with surface vision transformers (SiT) (Dahan et al. [022:] 2024), to support multimodal decoding of audio+video (movie) stimuli from 7T cortical fMRL. Importantly, this generalises to predicting new movie clips from new individuals (not used during training); using only a few seconds of fMRI; and is trained and tested using the HCP 7T watcciig dataset
[Van Essen al., 2013 Finn & Bandettini 2021), which collects many hours less data per indi-
vidual, than most recent decoding frameworks (Benchetrit et al. 2023 Défossez et al. 2023 Scotti
et al. 2024 2023 Ozcelik & VanRullen 2023). Achieving this involved engineering contributions
for tri-modal CLIP alignment of audio, video stimuli with fMRL. Benefits extend to reconstruction, where our embeddings may be used to reconstruct movie frames, from the brain activity of unseen subjects, that reliably decode the semantic content of scenes. Results show that joint encoding of au- dio/video stimuli and fMRI inserts complementary information that allows better decoding between any pair of these modalities. Furthermore, visualisation of self-attention suggests the model attends to functional brain networks audio-visual information. This opens the door to the future precision simulation (Digital Twins) of human brain function in response to unseen stimuli and tasks.
2 RELATED WORKS
Classic approaches to fMRI decoding, classify stimuli from the temporal dynamics of vox- els/vertices extracted from visual regions of the human brain. Most often linear or non-linear re- gression is used; with recent models leveraging the power of generative or foundational AI to learn rich encodings of stimuli, which are subsequently matched to patterns of brain activity through use
of CLIP contrastive learning (Benchetrit et al. 2023 Délossez al. 2023 Scotti al. 2024 2023 techniques
Ozcelik & VanRullen 2023). Often methods are complemented with "hyper-alignmen"
that seek to map all data to a space in which brain activations overlap across individuals (and so may compared) (Haxby et al. 2020 Thual et al. (223);; however, in practice, this is ill ill-posed which has meant that most decoding frameworks (Benchetrit et al. 2023 Défossez et al., 2023) (Sotti et al,, 2024 2023 Ozcelik& & VanRullen, 2023) are trained and tested within the same brain, using densely sampled datasets (Allen et al., 2021| Hebart et al. 2023) that collect tens of hours of recordings for each subject. Similarly, approaches to iuter-subject generalisation (Scotti al. 2024| Thual et al., 2023] introduce costly alignment steps that require 2 1 hour of recording for each test subject.
Learning-basee based frameworks support transformation equivariant modelling of images and, such, offer improved potential for generalisation. However, the cortex is a highly curved manifold with activations best modelled patterns across surface (Glasser al., 201 al Coalson al., 2018). This presents challenges due to the difficulty in translating convolutional operations to non- Euclidean domains that lack global coordinate system (Bronstein al., (02I]; resulting in frame- works that perform sub-optimally on cortical phenotyping and IMRI decoding tasks (Fawaz et al.)
2021
Gu al. 2022).
Recent work on SiTs (Dahan et al. 2022) has indicated that vision transformers robustly outperform surface convolutional neural networks (CNNs) across a range of cortical phenotyping tasks, while offering inherent interpretability through visualisation of self-attention. Moreover, careful adaption of the video masked autoencoder pre-training (Tong et al., [022] Feichtenhofer et al. 2022) to sur- face domains was shown to encode sufficient rich representations of cortical dynamics to decode cognitive traits (Dahan et al. 2024).
due noise and limited temporal resolution
3

---

Published as a conference paper at ICLR 2025
3 METHODS
In what follows, we consider cortical fMRI signals as functions in space and time S(v, 1) defined on spherical mesh (v E Va) corresponding to a 6th-order icosahedron: Ie (V6, Fs). with |Vel = 40962 vertices and [Fll = 81920 faces, see Figure A The objective of our framework is to use SiTs to encode the spatio-temporal dynamics of the signal as it evolves. This is imple- mented through video surface-masker autoencoder (vsMAE) self-supervision, Decoding is then implemented through aligning representations learnt from the vsMAE (Fig ( with audio and vi- sual representations (learnt from wav2vec (Baevski et al. 2020) and videoMAE (Tong et al. 2022 using CLIP contrastive learning (Fig DD). At inference time, this supports the retrieval of any one modality from each of the others.
3.1
BASE ARCHITECTURES
Cortical surface analysis Fischl 2012). is now a very common processing stage for most neuroimaging data collections, involving the fitting of tessellated meshes to the inner and outer boundaries of the cortex, followed by inflation and projection to a sphere (Figure []]. Cortical imaging features, such as fMRI are projected onto the surface through ribbon-constrained weighted averaging (Glasser et al., 2013). The SiT (Dahan et al.] 2022) leverages this simplified spherical domain to patch cortical imaging data using regularly tessellated icosahedral meshes. First imaging features are resampled to a high-resolution 46 = (Vo, Fb) mesh (with [V6] = 40962 vertices and F& 81920 faces; these features are then patched with the faces of low- resouutonn icosphere (typically Iz. with |F&| = 1280) (Fig B) partitioning the cortical features into a sequence of = |F/ non-overlapping triangular patches: P = {p' .pp..p N (with p' C Vo. Ip' = 45). Features within each patch are then concatenated across channels (C) flattened and projected, with trainable linear layer, into a set of D-dimensional input tokens to produce an initial sequence: X = [Xf...,
SiT:
R Sine-cosine positional embeddings, Epss {E J.V are then added to each of the tokens to encode patch location within the sequence: x² = [X] + Ei,... X& + Ex]. This initial sequence x0) is then processed by L consecutive transformer encoder blocks of H Multi-Head Self-Artention (MHSA) and Feed Forward Network (FFN) layers, with residual layers in-between, resulting in an output sequence of fMRI token embeddings (AyMRI E DN As with classic vision transformers (Dosovitskiy et al. 2020) the objective is to model long-range co- occurrences of spatial structure in the surface imaging data (as self-attention between tokens), which should confer sufficient image understanding to perform any given supervised learning task.
vsMAE: Transformers are powerful learning models but their lack of inductive biases present challenges when training on limited data. In Dnnanh et al. al. (2024). self-supervision pre-training task was proposed that extends the concept of midko-masked autoencoders (Tong al., 2022 Feicht- enhofer et al., 2022) to the surface. This frames self-supervision as an auto-encoding task, where unmasked tokens, from T input fMRT frames, first randomly selected from the set of all available patches according to a masking ratio p. Each token then represents a 'tube' of cortical data patched SiT encoder (B./R()) which compresses each spatio- temporal token through its linear layer, and then passes the resulting sequence of tokens through each transformer encoder block. Next, random embeddings are added into the encoder's latent em- beddings sequence- in place of the masked tokens restoring the sequence to its original length. Positional embeddings are added to encode spatial information and the resulting sequence is fed into the SiT decoder (MMRI ). The last layer performs a linear projection to restore the input patch resolution (T' x (p')). Following He et al. (2021). the vsMAE is optimised by calculating the mean square error (MSE) between the masked input feature patches and their reconstructed versions only. The pre-trained vsMAE encoder /M/RI then forms the basis of the proposed decoding framework exc
in space and time. These
passed
(Fig l..
3.2 DECODING NETWORK
Taking learnt representations of the fMRI from the pre-trained vsMAE encoder (XYMRR), the next objective is to align these to video (V) and audio (A)) representations, respectively noted Xy E R Ny and X' E NAXDAA learnt from pre-trained videoMAE (Togg et al. (022) and
wav2vec2.0 models (Baevski et al. 2020) ( eeee and Φ-4 in Fig 1 & F), using CLIP contrastive
4

---

Published as a conference paper at ICLR 2025
learning (Radford et al. 2021). Details about audio-visure stimuli processing and embeddings traction are provided in Appendix[B. B.3
Multimodal mappers: Since each unimodal model outputs a sequence with different token length, it is necessary to first compress all representations to vectors of the same length such that they can be directly compared (Fig D). This is performed with multimodal mappers sgmod, similar to (Najennkokk et al., 2023), that project each unimodal sequence of tokens through two linear layers, with GeLU activation, dropout and residual connections, before averaging the sequence to output a vector of equivalent resolution (DCLIP) for each modality: such that yfMRI = =1 ffaR (XęMRt), UA 1 (Xx) and yv = S8 (R and yf MR, yv, U.A E R DeLIP.
Alignment: Tri-modal CLIP alignment is then achieved by sampling, for each batch, exactly one positive triplet and M negative triplets. A positive triplet consists of fMRI, audio, and video data from the same 3s movie clip, while negative triplets are sampled from different 3s movie clips. Next, cosine similarities (i.,) = (v'. v/²) are calculated between pairs of modalities (a and b), which are then converted into probabilities through applying softmax function: Pa,
((,.). Ei-1 (x.p=.,,((.A))1) (here, T is a temperature hyperparameter that scales the logits). The CLIP loss from modality to b (noted La-s) is then calculated using cross- entsopy to push together the embeddings of positive samples and push apart the embeddings of negative samples, such that:
M Ei=1 log j)) To perform alignment across three modalities simultaneously, we average the losses calculated between all pairs of audio/video/IMRI in the batch:
La-tb 1 M
-v + Ly->ism + Lismi > + LA-MMR LA-VV + Ly~+a/66
Then, at inference time, and for each given modality a, the model can be used to rank the samples that are most likely be aligned with modality (highest probability) from list of available stimuli.
Using the methodology proposed by (Ozcelik & VanRullen 2023 (and em- ployed in (Benchetrit et al. 2023)), we developed a two-stage framework to reconstruct visual stimuli from cortical fMRI data. In the first stage, regression model (reg,) ) is trained to predict visual latent representations (extracted from a pre-trained VDVAE model (Child 202I)) from fMRI signals. In the second stage, separate regression model (reg,) is trained to map IMRI CLIP embed- dings (U)MRI) ) to video CLIP embeddings (tv)). This model is trained on one fMRI session for one training subject. During inference, test fMRI signals (from new subject and new movie scene) first processed by reg, to generate latent visual representations and decoded by the VDVAE model to produce low-resolution image reconstructions. These low-resolution images, along with multimodal guidance from reg2, are input into a pre-trained Versatile Diffusion (VD) model (Rombach et al., 2022) to generate the final high-resolution visual reconstructions, see Figures C.10 and
Reconstruction:
4 EXPERIMENTAL METHODS
4.1 DATASET
In this paper, stimuli and accompanying brain recordings were taken from 174 participants, aged 29.4 years (68 male and 106 female) who were scanned as part of the HCP 7T movie-watching experiment (Van Essen al. 2013 Finn & Bandettini 202T). Participants underwent 4 recording sessions, each lasting ~ 15 minutes, during which time they were shown series of movie scenes from independen//Hollywodd movies (1-4.3 mins in length). These movie scenes were interleaved with rest periods of 20 seconds, where participants were told to fixate on cross on blank screen (Fig2). 2). For each session, audio was delivered via earbuds, and movie files were cropped and zoomed from their original 16:9 aspect ratio (AR) to a 1024 x 720, 14.22: 10 AR to fit the projector screen. All Hollywood movie scenes were prepared and published by (Cutting et al. 2012).
7 Tesla Siemens Magnetom scanner, using a gradient- echo EPI sequence with repetition time (TR) = 1s, echo time (TE) = 22.2ms, and spatial resolution =1.6mm3 (Finn& & Bandettini 2021). These volumes were motion- and iestortion-correcte high- pass filtered, and boundary-base based aligned (Greve & Fischll 2009) to T1- and 12-weighted sMRI acquired at (). mmm resolution, by the HCP's custom 3T Siemens Skyra (Glasser al. 2013).
During sessions, fMRI was acquired
5

---

Published as a conference paper at ICLR 2025
Confound signals were then regressed from the data, where these correspond to 24 measured msotion parameters (at each timepoint) and a series of data-driven noise components derived from FIX ICA (Salimi- Khorshidi et al. 2014)).
All data was processed through the HCP minimal processing surface pipelines GGlasser et al. 2013) (adapted from FreeSurfer (Fischl 2012)). This maps fMRI timeseries onto tessellated meshes rep- resenting each individual's cortical anatomy. Data is then mapped to the sphere and aligned using MSMAll functional alignment, where this has been shown to considerably improve the overlap of functional activations across brains, relative to volumetric or cortical shape-based alignment (Robin-
son et al., 2014, 2018 Glasser et al. [0166a 2013 Smith al., 2013 Coalson et al., 2018). Features are resampled from native resolution (59292 vertices) to [6]. using adaptive barycentric interpola-
tion (Robinson et al. 2018).
4.2 TRAINING
Subjects were partitioned into train/validaaiion/ess splits of size 124/25/25, while stratifying sex and age distribution across splits, with fMRT from left and right hemispheres treated as independent samples but placed in the same split. This corresponds to 992 training, 200 validation and 200 lesting samples. We then divide the movie data into a series of non-overlapping 3s movie clips: corresponding to 16 frames of movie stimuli and 3 frames from the cortical IMRI, where this was sampled with temporal lag of 6 seconds to account for the haemodynamic response (Huth et al., 2016) (Appendix C.8).
The SiT backbone, forming & RRJ follows the standard structure of a DeiT-small (Tovrron et al,, 2020), which extends (Dosovitskiy et al., 2020) to more efficient ViT architectures. For all training phases (vsMAE pre-training and tri-l CLIP alignment), the Adam (Loshchilov & Hutter
2019) optimisation was used with LR = 3e - and cosine decay. Distributed training was imple- mented in all experiments with a batch size of 64 (per GPU) for the vsMAE pre-training task. For the tri-modal CLIP training, batch size was maximised across all GPU instances to 256 by implement- ing aggregation across instanced All experiments were on 4 NVIDIA V100 GPUs (32 GB of memory). Video and audio encoders (&)
$ Dethe) were kept frozen for all experiments. Multimodal mappers (ff f/', f6") were trained in all experiments. During the tri-modal CLIP training, we investigated various training regimes for the pre-trained vsMAA encoder: (1) training from scratch, (2) keeping the SiT encoder frozen after vsMAE pre-training or (3) fine-tuning from pre-trained weights. To evaluate the impact of different modalities during the CLIP-alignment training, we also investigated the training of two modalities only (e.g. fMRI, V) or the three modalities altogether (fMRI, V A). Ablation experiment, for (1) & (2) are shown in Table C.I
4.3 INFERENCE
At test time, models were evaluated based on their retrieval and reconstruction abilities, where in- ference of modality a from modality b is defined as - b (e.g. faR - V). In practice, this is calculated from sampling M test samples, including one positive (correct) pair and M - 1 negative (mismatched) pairs. These are passed to the CLIP model, which outputs a (softmax) probability that each of these pairs is match. Results are then ranked and performance is reported from top-K K accuracy i.e. whether the true pair ranks within the top K samples, as per recent works (Thual al. 2023 Ozcelik & VanRullen 2023 Scotti et al. 2023 2024| Benchetrit et al.) 2023, Défossez et al, 2023). The number of clips tested is adapted for each pair of modalities to account for different noise levels applying M = 64 for video and M = 32 for audio testing (audio samples being noisier). Two types of negative sampling procedures were used: soft-negative- where negative pairs are sampled only from different movies to the positive sample; and hard-negative sampling - where negatives were sampled only from within the same movie (and thus share semantic content). Impor- lantly, we used a sampling buffer around the positive sample, ensuring that no negatives were taken from within #:3s of a positive sample. The choice of 3s to define movie clips corresponds to the average duration of movies 'shots" (see more details in Appendix A.2). Main retrieval results are
provided in Table I and Figure C.2 and additional results in Tables C.I & and Figure C.3
Implementation provided in httppss//ggithhb....//ttttttiiie

---

Published as a conference paper at ICLR 2025
(a)
REST
20
20
3-4mm
20
mm
20
(b)
()) Experiment gener
(1) Experiment generaliation
(8)) Experiment 3: generalication
subjects
move scercc
subjects now movis accnnss
Subjocts
Subjett
Subjeds
Training
Testing
Training
Teasing
Training
Testing
All movie scenes
All movie scenes
Movie scene A
Movie scene
Movie scene A
Movie scena
Figure 2: (a) Each movie- session (MOVIE1-4) is composed of movie scenes extracted from different movies (ranging in total from 1.4 to 4.3mn) and interleaved with 20s rest inter- vals. Movie scenes further divided into 38 movie clips for training and inference processes. (b) Overview of the three experimental setups: (i) Experiment I: Subjects are divided into training and testing groups; training involves all movie clips while testing validates whether movie clips that were seen during training but using brain activations of new subjects from the test set. (ii) Experiment 2: utilises all subjects for both training and testing, but only the first half of each movie as training; we then validate on whether we can decode new clips from movie scenes that were not seen during training. (iii) Experiment 3: Training limited subset of subjects (as in (i)) and only the first half of each movie (as in (ii)); the model is then tested on decoding new movie clips- - from the last half all movies from the brain activations of new subjects not seen during training. Figure C.I further clarifies the multilevel sampling terminology.
decode
4.4 EVALUATION
The objective of this paper is to demonstrate that it is possible to train encoding and decoding frameworks that generalise to (i) new subjects (Experiment 1y; (ii) new movie scenes (Experiment 2); and (iii) new movie scenes within new subjects (Experiment 3). The experimental setup is described in F
Beyond evaluation of retrieval performance, we also consider: (1) how well self-attention maps, encoded by the vsMAE, reflect the current understanding of the underlying cognitive processes; (2) if tri-modal alignment allows stimuli reconstructions from fMRI signals. We, therefore, visualise attention maps by projecting them back to the cortical surface (Fig4). This is achieved by extract- ing the self-attention weeihts matrix from all layers, passing these through softmax operations; then CarlowinnCCroonn slicing to retain only weights associated with the output [CI.S] token (following Caron et al. (2021) ) These weights are then interpolated back Is resolution by assigning all vertices, for each corti- cal patch, the attention weight of the corresponding token in the sequence. For the video-frame reconstruction, we adapt the code from Ozcelik VanRullen (2023) to the CLIP/latent encoding resolutions of our architecture; applying 50 DDIM steps with a strength of 0.75.
To evaluate the contribution of the SiT over and above the impact of contrastive learning, compare against Ridge regression models, taking inspiration from Ozcelik & VanRullen (223) Since all data has been MSMall (functionally) aligned across subjects, this represents an extremely robust baseline.
Baseline:
7

---

Published as a conference paper at ICLR 2025
tap1
top5
top28
top30
topez
(positive
b4)
Text subject
Test subject
Figure 3: Video retrieval for Experiment generalisation to new movie scenes and new subjects from: (a) soft-negative sampling: here, the reference movie clip correctly retrieved topl and top-ranked movie clips all depict human faces; (b) hard-negative sampling: here the top-ranked movie clips all correspond dialogue scenes.
RESULTS
Generalisation to new subjects (i) Results in Table Table@repone report the top-1 and top-10 performance for Experiment I (described in Fig(2 in two inference directions fARI - V, fMRI - A. In both cases, the best retrieval performance is achieved when training on all three modalities, demonstrating the complementary contributions of video and audio for decoding (see also Appendix: Table C.1 for the inverse retrieval results). In all cases, performance increases considerably over random and ridge regression baselines, even for CLIP alignment of only two modalities. Table [ summarises results for hard-negative sampling and decoding only but comparable improvements are seen for soft-negative sampling, and inverse retrieval (V' fMR and A + fMR)), which may be seen as encoding brain activations from audio/video stimuli. For more results, please refer to Appendix C Figure C.7 and Figure C.8
Table C.I
Generalising to movie scenes and new subjects (ii) & (iii) Results for Ex- periment and Experiment 3 for soft- negative sampling are reported in Figure C.2. and demonstrates that the SIM frame- works clearly generalises to new movie scenes and new subjects, compared to base- lines. Visualisation of retrieved clips (Fig- ure 3. for two test subjects, and for both hard-negative and soft-negative sampling. shows that our model meaningfully ranks movie clips with similar contents together. The full tables of results are available in Appendix C (TTble C.2 and CC3 in all cases the proposed model (fMRI - V) strongly outperforms baselines.
Generalisation new subjects (Experiment I). tested on all movies. Bold corresponds to the best- performing model for each inference modality. Re-
Table 1:
sults (in %) with and 95% conf. interval (CI).
Training modalities (CLIP)
clip retrieval
Inference Modality
FMRI enC
top-1 C
top-10
Random
X fMRI, V
3.74 0.5
19.4+-0.9
Ridge
15.6+
64.1+11.6
[MRI
SiT (ours)
fMRII V
64.7422.
95.6+
SiT (ours) fMRI, V, A
76.8+
94.2+
Random
X
3.140.4
30.1+1.1
Ridge fMRI, A
3.2+0.4
30.5+112
[MRI - A
SiT (ours) fARRI A
19.9+1..
62.5+1.7
SiT (ours)
fMnu, V, A
56.6::2.5
86.9+22
Interpretation of self-attention maps Visualisation of attention maps, corre-
sponding cortical fMRI, averaged across all test subjects, from 3s movie clip involving dia-
8

---

Published as a conference paper at ICLR 2025
Attention Head
Attention Head 2
Attention Head 3
Average ateenton
Broudh
Barennssery & Heter coriso
Vual cora
Attention Head
Attention Head 5
Attention Head
STD attontion
Stimuli
Frame
Frame 2
Frame 3
Figure 4: Average attention maps for each attention head, extracted from the SiT encoder ( (R from 3s movie clip of dialogue scene during Ocean's 's Eleven. Maps averaged across test sub- jects (in Experiment I). Brain regions of importance for the movie- movie-watching watching stimuli are highlighted and annotated based the HCP multimodal parcellation (Glasser al. (016b)) Scene files will be made available on BALSA. Comparison with functional networks in Appendix C.33
ene
logue, are shown in Figure Results are shown averaged within the individual HCP multimodal areal parcellation (Glasser et al. 2016b). This is done to support comparisons between cortical areas and their reported functions (see Tables 1-3 supplementary anatomical results (Glasser et al. 2016b)). Results shown for each attention head separately, also display the mean and variance calculated across all heads, for all test subjects.
Systematic comparison of the activation patterns from each attention head, against well-validated topographic maps functional networks Yeo et al. 2011 Margulies et al. 2016) reveals their spe- cialisation into sensorimotor, visual, and auditory cortices. A correlation analysis against Margulies' gradient-based maps (Margulies et al. 2016) shows that Gradient 2 is the highest correlated with all attention heads, pointing to dominance of attention within visual and sensorimottr//addiory cortices. Closer examination shows that attention heads to 4 specialise in sensorimotor/auditory processing, while heads 5 and specialise in visual processing. Similarly, comparison with the functional networks identified by Yeo et al. (Yeo et al. 2009) shows that attention heads I to highly correlate with Network 2 (sensorimotor and auditory). whereas heads and correlate with Network 1 (vi- sual). Margulies's 's gradient maps (Margulies et al.) 2016) and Yeo's functional networks (eo et al.) 2009 are provided in Tables
C.5 for qualitative comparison.
(a) Experiment subjects
Experiment subjects
Recemiracted
Firumes
Figure 5: Video-frame reconstruction results from fMRI embeddings after tri-modal CLIP align- ment. Following Ozcelik & VanRullen (2023). the reconstruction pipeline was trained
train- ing subject and tested on a new subject (a) on the same movie scenes as used for training and (b) on new movie scenes. Reconstructions realistic, preserving most of the semantie information from the original frame and, importantly, generalise to new movie scenes and new subjects.

---

Published as a conference paper at ICLR 2025
6 DISCUSSION
Previous studies have shown that modern Al frameworks, such as CNNs and transformers, model
natural stimuli in ways that parallel human cognitive processing (Millet al. 2022 Antonello & Huth 2024;
Caucheteux et al. 2023 Kriegeskorte 2015) and that this can be harnessed to decode
auditory and visual stimuli from human brain recordings (Benchetrit et al.. 2023 Défossez et al.)
2023 Scotti et al. 2024 2023 Ozcelik & VanRullen 2023; Gu et al. 2022
Lindsay |021] Thomas
ct al. 2022: Thual et al. 2023 Wen et 2018| Yamins & DiCarlo 2016). While powerful, these
approaches lack any model of individual cortical areal topography from which to simulate unseen stimuli. In this paper we take an important step in this direction by showing that SiT encoding of IMRI ppatial-autocorrelationn can allow for CLIP decoding of new movie scenes, from subjects that the model was not trained on, Moreover, vision transformers are inherently interpretable allowing us to visualise the patterns of self-attention encoded by the model. Our results suggest that each attention head may be modelling different visual and semantic concepts. Thus far, this has only been assessed a global level by comparing average patterns of attention against state-of-the-art models of functional organisation including gradient maps (Margulies et al. 2016), functional connectivity networks (Yeo et al.) (200) and the HCP multimodal parcellation (Glasser al. (0T6a)). We are yet to explore whether self-attention varies mcaningfully across individuals, including whether maps are predictive of behavioural or cognitive traits (Finn & Bandettini] 202T).
One notable current limitation with the current SiT architecture is how its computations scale with resolution. Increasing sampling of cortical patching by a single resolution level (from 13 to I4) increases the complexity of self-attention operations 16x. As this trade-off is critical for contrastive learning, exploring lighter forms of attention computation could help with scaling up the model. Another important consideration is the current lack of any model of sub-cortical fMRI, despite the known involvement of deep grey structures such as the Lateral Geniculate Nucleus (LGN) in vision (Ghodrati et al. (2017). In future this could be addressed by adding tokens for sub-cortical structures. Similarly, our study focuses on matching modalities based solely on 3s movie clips. While this represents attempt modelling temporal dynamics, increasing the length of samples might allow for modelling of more complex brain processes, such as memory and attention.
In doing it may be important to consider expanding the dataset to include wider range of audio- visual stimuli. While the HCP 7T mooiii-atthiing dataset is an incredible resource, collected across large number of subjects and crucial to evaluate generalisation properties, it comprises a restricted set of audio-visual stimuli, collected from very different styles of movies. This constrains the amount of information that can be extracted and learnt by our model, meaning that we were unable to effectively test whether the model would generalise to completely different movies. In particular, this can also limits the potential for stimuli reconstruction (Figure(5) as the variety of visual signals is limited. Alternative datasets, which are richer in the amount of data collected per subject, include the 'Friends' fMRI dataset (Boyle J.A. 2020) and the 'Narratives' dataset (Nastase et al. 202T).
When considering the broader impact of such technologies, our interest is in tailoring brain- computer interfaces and neurofeedhack therapies to individual brains, in which the needs and abil- ities of individual patients must be taken into account. Under these conditions, development of decoding models that generalise across brains are of vital importance, as they would allow for the sampling different experimental conditions across participants, requiring far less data collection for each new individual, while allowing models to simulate outside their training regime i.e. respond to novel situations. While, practical deployment of such healthcare models remains long way off, it is important to be mindful of longterm potential ethical impacts of models that might generalise the de- coding of human thought from brain activations; not least because all freely available open datasets are derived from healthy (largely Caucasian) controls, which risk model bias and poor generalisation to patient and minority groups
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
Simon Dahan would like acknowledge funding from the EPSRC Centre for Doctoral Training in Smart Medical Imaging (EP/S022104/1), This project also made use of time on Tier 2 HPC facilities JADE2, funded by EPSRC (EP/P020275/1).
REFERENCES
Emily J Allen, Ghislain St-Yves, Yihan Wu, Jesse L Breedlove, Jacob S Prince, Logan T Dow- dle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, Benjamin Hutchinson, Thomas Naselaris, and Kendrick Kay. A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence. Nat. Neurosci. December 2021.
Richard Antonello and Alexander Huth. Predictive coding or just feature discovery? an alternative account of why language models fit brain data. Neurobiol Lang (Camb), 5(1):64-79, April 2024.
Alexei Baevski, Hemry Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A frame- work for self-supervised learning of speech representations, 2020.
Yohann Benchetrit, Hubert Banville, and Jean-Rémi King. Brain decoding: toward real-time recon- struction of visual perception. arXiv preprint arXiv:2310. 19812, 2023.
Janine Diane Bijsterbosch, Mark W Woolrich, Matthew Glasser, Emma C Robinson, Christian Beckmann, David C Van Essen, Samuel J Harrison, and Stephen M Smith. The relationship between spatial configuration and functional connectivity of brain regions. Elife, 7, February 2018.
Pinsard B. Boyle J.A. The courtois project on neuronal modelling 2020 data release. In OHBM, 2020. http::/
www.rnnnuooooo. ca/caa/eeaa hte/en/lateese DDATASETT. html
Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Veličković. Geometric deep learn- ing: Grids, groups, graphs, geodesics, and gauges, 2021. URL
abs/
2204.13477
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers, 2021. URL https:
1aaxxiv.orr//as//1000.11 14294
Charlotte Caucheteux, Alexandre Gramfort, and Yean-Rémi King. Model-based analysis of brain activity reveals the hicrarchy of language in 305 subjects. arXiv preprint arXiv:2110.06078, 2021.
Charlotte Caucheteux, Alexandre Gramfort, and Jean-Rémi King. Evidence of predictive coding hierarchy in the human brain listening to speech. Nature human behaviour, 7(3):430444, 2023.
Rewon Child. Very deep vaes generalize autoregressive models and can outperform them on images, 2021.
TS Coalson, DC Van Essen, and MF Glasser. The impact of traditional neuroimaging methods on the spatial localization of cortical areas. Proceedings of the National Academy of Sciences the United States of America, 115 115(2)) 2018. doi: 10.1073/pmas. 1801582115. URL Epub 2018 Jun 20.
https:// //w::///wwww.. pnas org/ http://nnnnettnnnnnneeee https//wwwww.ppaaooooccooootttt/
James E. Cutting, Kaitlin L. Brunick, and Ayse Candan. Perceiving event dynamics and parsing hollywood films. Journal of Experimental Psychology: Human Perception and Performance, 38(6):146 1490, December 2012. doi: 10.1037/a0027737. | http::ttss/ssc //op::://do.... org/l0. 1037/a0027737 Epub 2012 Mar 26.
Simon Dahan, Abdulah Fawaz, Logan Z. J. Williams, Chunhui Yang, Timothy S. Coalson, Matthew F. Glasser, A. David Edwards, Daniel Rueckert, and Emma C. Robinson. Surface vi- sion transformers: Attention-base based modelling applied to cortical analysis. In Proceedings of The 5th International Conference Medical Imaging with Deep Learning, Proceedings of Machine Learning Research, pp. 282-303, PMLR, 06-08 Jul 2022. URL https: :tttta::/prooe h/tpro/cecceeegg......
m www.press/v112/daaaaae html
11

---

Published as a conference paper at ICLR 2025
Simon Dahan, Logan Zane John Williams, Yourong Guo, Daniel Rueckert, and Emma Claire Robin- son. Spatio-temporal encoding of brain dynamics with surface masked autoencoders. In Proceed- ings of The 7nd International Conference Medical Imaging with Deep Learning, volume 250 of Proceedings of Machine Learning Research, pp. 306-325, PMLR, 03-05 Jul 2024. URL. htttps://procent
ml https//mmm..eeeee///////aaaaaaa
Alexandre Défossez, Charlotte Caucheteux, Jérémy Rapin, Ori Kabeli, and Jean-Rémi King, De- coding speech perception from non-invasive brain recordings. Nature Machine Intelligence, (10):1097-1107, 2023.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszko- reit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition
scale. CoRR, abs/2010. 1929, 2020. / 200p://aaa//000..112 11929
Ian Ellwood. Short-term hebbian learning can implement transformer-like attention. PLOS Com- putational Biology, 20(1):: 10143,,, 2024.
Abdulah Fawaz, Logan Z. J. Williams, Amir Alansary, Cher Bass, Karthik Gopinath, Mariana da Silva, Simon Dahan, Chris Adamson, Bonnie Alexander, Deanne Thompson, Gareth Ball, Christian Desrosiers, Hervé Lombaert, Daniel Rueckert, A. David Edwards, and Emma C. Robin- son. Benchmarking geometric deep learning for cortical segmentation and neurodevelopmen- tal phenotype prediction. bioRxiv, 2021. doi: 10.1101/2021.12.01.444 470730. URL https: 12, 01. 01.44733337
//www www.w.biorxiv
Christoph Feichtenhofer, Haoqi Fan, Yanghao Li, and Kaiming He. Masked autoencoders as spa- tiotemporal learners, 2022.
Emily S Finn and Peter A Bandettini. Movie-watching outperforms rest for functional connectivity-
based prediction of behavior. Neuroimage, 235:117963, 2021. doi: (0.1016/.neuurrimaee2222 117963. Epub 2021 Apr 2.
Bruce Fischl. Freesurfer.
Neurolmage, 62.774-781, 8 2012. ISSN 1095-9572. 10.1016/J 2AW.GGENEEUOOMMA.2222.200.00000. 222485737
doi:
URL bttps://pubmed. ncbi nlm. gov/
Masoud Ghodrati, Seyed-Mahdi Chaligh-Razavi and Sidney R Lehky. Towards building a more complex view of the lateral geniculate nucleus: Recent advances in understanding role. Progress in Neurobiology, 156:214-255 2017.
Matthew Glasser, Stamatios Sotirppoulos, J. Wilson, Timothy Coalson, Bruce Fischl, Jesper Ander- sson, Junqian Xu, Saad Jbabdi, Matthew Webster, Jonathan Polimeni, Van DC, and Mark Jenkin- son. The minimal preprocessing pipelines for the human connectome project. Neurolmage, 80: 105, 10 2013. doi: 10.10166/. neuromaae.rr 2013.041. 127.
Matthew F Glasser, Timothy S Coalson, Emma C Robinson, Carl Hacker, John Harwell, Essa Yacoub, Kamil Ugurbil, Jesper Andersson, Christian F Beckmann, Mark Jenkinson, et al. A multi-modal parcellation of human cerebral cortex. Nature, 536(7615) 2016a.
Matthew Glasser, Stephen M Smith, Daniel Marcus, Jesper LR Andersson, Edward Auerbach, Timothy EJ Behrens, Timothy S Coalson, Michael Harms, Mark Jenkinson, Steen Moeller, et al. The human connectome project's neuroimaging approach. Nature neuroscience, 19(9): 1185-117, 2016b.
Evan M Gordon, Timothy O Laumann, Babatunde Adeyemo, Adrian W Gilmore, Steven M Nel- son, Nico UF Dosenbach, and Steven E Petersen. Individual-specific features of brain systems identified with resting state functional correlations. Neuroimage, 146:918 93999999999..11 2017a
Evan M Gordon, Timothy O Laumann, Adrian W Gilmore, Dillan J Newbold, Deanna J Greene, Jef- frey J Berg, Mario Ortega, Catherine Hoyt-Drazen, Caterina Gratton, Haoxin Sun, al. Precision functional mapping of individual human brains. Newron, 95(4):791-807, (00079991807,1777779 2017b.
Douglas N Greve and Bruce Fischl. Accurate and robust brain image alignment using boundary- based registration. Neuroimage, 48(1):63-72, 2009.
12

---

Published as a conference paper at ICLR 2025
Zijin Gu, Keith Jamison, Amy Kuceyeski, and Mert Sabuneu. Decoding natural image stimuli from fmrr data with surface-basee convolutional network. arXiv preprint arXiv:2212.02409, 2022.
James V Haxby. Swaroop Guntupalli, Samuel A Nastase, and Ma Feilong. Hyperalignment Modeling shared information encoded in idiosyncratic cortical topographies. elife, 9:e56601, 2020.
Kaiming He, Xinlei Chen, Saining Xic, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders scalable vision learners, 2021.
Martin N Hebart, Oliver Contier, Lina Teichmann, Adam H Rockter, Charles Y Zheng, Alexis Kid- der, Anna Corriveau, Maryam Vaziri- Pashki,a and Chris Baker. THiNGG-data, a multimodal collection of large-scale datasets for investigating object representations in human brain and be- havior. Elife, 12, February 2023.
A. Huth, W. de Heer, T. Griffiths, et al. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature, 532:453 458, 2016. doi: 10038/aauure16337 URL https://d// Grg/10..038/natuue
7637
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The kinetics human action video dataset, 2017.
Dong Kyum Kim, Jea Kwon, Meeyoung Cha, and C Lee. Transformer as a hippocampal mem- ory consolidation model based on nmdar-inspired nonlinearity. Advances in Neural Information Processing Systems, 36, 2024.
Ru Kong, Jingwei Li, Csaba Orban, Mert R Sabuncu, Hesheng Liu, Alexander Schaefer, Nanbo Sun, Xi-Nian Zuo, Avram J Holmes, Simon Eickhoff, al. Spatial topography of individual- specific cortical networks predicts human cognition, personality, and emotion. Cerebral cortex, 29(6):2533 225111,,,,,,,,, 2019.
Nikolaus Kriegeskorte. Deep neural networks: a new framework for modeling biological vision and brain information processing. Annual review of vision science, 1:417-446, 2015.
Robert Leech, Reinder Vos De Wael, František Váša, Ting Xu, R Austin Benn, Robert Scholz, Ro- drigo M Braga, Michael Milham, Jessica Royer, Boris C Bernhardt, Emily J Jones, Elizabeth Jefferies, Daniel S Margulies, and Jonathan Smallwood. Variation in spatial dependencies across the cortical mantle discriminates the functional behaviour of primary and association cortex. Nat. Commun., 1411),5656, September 2023.
Grace W Lindsay. Convolutional neural networks model of the visual system: Past, present, and future. Journal of cognitive neumscience, 33(10):2017-2033 2021.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization, 2019.
Daniel S Margulies. Satrajit S Ghosh, Alexandros Goulas, Michat Falkiewicz, Julia M Huntenburg, Georg Langs, Gleb Bezgin, Simon B Eickhoff, Francisco Xavier Castellanos, Michael Petrides, Elizabeth Jefferies, and Jonathan Smallwood. Situating the default-mode network along prin- cipal gradient of macroscale cortical organization. Proceedings of the National Academy of Sci- ences, 113444))15774-1257, 2016. doi: 10. 10.1077/pnas... 16082222 Epub 2016 Oct 18.
Juliette Millet, Charlotte Caucheteux, Pierre Orhan, Yves Boubenec, Alexandre Gramfort, Ewan Dunbar, Christophe Pallier, and Jean-Remi King. Toward realistic model of speech processing in the brain with self-supervised learning, 2022. URL htttt:t//aaxiiii http://axxxxx...////oo//o/ wttp:org///bb///o/ss
01 685
Ivona Najdenkoska. Xiantong Zhen, and Marcel Worring. Meta learning bridge vision and lan- guage models for multimodal sew-shot learning, 2023.
Samuel A Nastase, Yun-Fei Liu, Hanna Hillman, Asieh Zadbood, Liat Hasenfratz, Neggin Ke- shavarzian, Janice Chen, Christopher J Honey, Yaara Yeshurun, Mor Regev, et al. The "narra- tives" fmri dataset for evaluating models of naturalistic language comprehension. Scientific data, 8(1):250, 2021.
13

---

Published as a conference paper at ICLR 2025
Furkan Ozcelik and Rufin VanRullen. Natural scene reconstruction from fmri signals using genera- tive latent diffusion. Scientific Reports, 13(1):1566, 2023.
Vassil Panayotov, Guoguo Chen, Daniel Povey, and Sanjeev Khudanpur. Librispeech: An asr corpus based on public domain audio books. In 2015 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 5206-5210, 2015. doi: 10.1 10.1109/ICASPP201..
7178964.
Usama Pervaiz, Diego Vidaurre, Chetan Gohil, Stephen M Smith, and Mark W Woolrich. Multi- dynamic modelling reveals strongly time-varying resting fmri correlations. Medical image anal- ysis. 77:102366, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision, 2021.
Emma C. Robinson, Saad Jbabdi, Matthew F. Glasser, Jesper Andersson, Gregory C. Burgess, Michael P. Harms, Stephen M. Smith, David C. Van Essen, and Mark Jenkinson. MSM: new flexible framework for Multimodal Surface Matching. Neurolmage, 100:414 oct 2014. ISSN 1095-9572. doi: 10.1011/J WEW.NNEEEAAAAE.11...0000 nlm. nih. gov /29333342
URL https: http:/::/ppmmmddd.eecc ncb...
Emma C. Robinson, Kara Garcia, Matthew F. Glasser, Zhengdao Chen, Timothy S. Coalson, Anto- nios Makropoulos, Jelena Bozek, Robert Wright, Andreas Schuh, Matthew Webster, Jana Hutter, Anthony Price, Lucilio Cordero Grande, Emer Hughes, Nora Tusor, Philip V. Bayly, David C. Van Essen, Stephen M. Smith, A. David Edwards, Joseph Hajnal, Mark Jenkinson, Ben Glocker, and Daniel Rueckert. Multimodal surface matching with higher-order smoothness constraints. doi: 10. 10.100//JNUUROIAAG...11.. 10.037. URL https: http::ttts:/plttt //pbmee. ncbi. nlm. nih. http://op////./g//
Neurolmage, 167:453-465, feb 2018 ISSN 1095-9572.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models, 2022. URL
abs/2112. 10752
Gholamreza Halimi-Khooshiid Gwenaëlle Douaud, Christian F Beckmann, Matthew F Glasser, Lu- dovica Griffanti, and Stephen M Smith. Automatic denoising of functional mri data: combining independent component analysis and hicrarchical fusion of classifiers. Neuroimage, 90:449-468, 2014.
Paul S Scotti, Atmadeep Banerjee, Goode, Stepan Shabalin, A Nguyen, E Cohen, Aidan Dempster, Nathalie Verlinde, Elad Yundler, David Weisberg, K Norman, and T Abraham. Reconstructing the mind's eye: fMRI-to-image with contrastive learning and diffusion priors. Adv. Neural Inf.
Process. Syst. abs/2305. 18274, May 2023.
Paul Scotti, Mihir Tripathy, Cesar Kadir Torrico Villanueva, Reese Kneeland, Tong Chen, Ashutosh Narang, Charan Santhirasegaran, Jonathan Xu, Thomas Naselaris, Kenneth A Norman, et al. Mindeye2: Shared-subject models enable fmri-to-image with hour of data. arXiv preprint arXiv:2403.11207, 2024.
Maxwell Shinn, Amber Hu, Laurel Turner, Stephanie Noble, Katrin H Preller, Jie Lisa Ji, Flora Moujaes, Sophie Achard, Dustin Scheinost, R Todd Constable, John H Krystal, Franz X Vollen- weider, Daeyeol Lee, Alan Anticevic, Edward Bullmore, and John D Murray. Functional brain networks reflect spatial and temporal autocorrelation. Nat. Neurosci, 26(5):867-878 May 2023.
Stephen M Smith, Diego Vidaurre, Christian F Beckmann, Matthew F Glasser, Mark Jenkinson, Karla L Miller, Thomas E Nichols, Emma C Robinson, Gholamreza Salimi-Khorshidi, Mark W Woolrich, al. Functional connectomics from resting-state fmri. Trends in cognitive sciences, 17(12):666-68) 2013.
Jerry Tang, Meng Du, Vy Vo, Vasudev Lal, and Alexander Huth. Brain encoding models based on multimodal transformers can transfer across language and vision. Advances in Neural Information Processing Systems, 36, 2024.
14

---

Published as a conference paper at ICLR 2025
Armin Thomas, Christopher Ré, and Russell Poldrack. Self-supervised learning of brain dynamics from broad neuroimaging data. Advances neural information processing systems, 35:21255- 21269, 2022.
Alexis Thual, Yohann Benchetrit, Felix Geilert, Jérémy Rapin, Iurii Makarov, Hubert Banville, and Jean-Rémi King. Aligning brain functions boosts the decoding of visual semantics in novel sub- jects. arXiv preprint arXiv:2312.06467 2023.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang- Videomae: Masked autoencoders are data- efficient learners for self-supervised video pre-training, 2022.
Hugo Touvron, Matthieu Cord, Matthijs Douze, Francisco Massa, Alexandre Sablayrolles, and
Hervé Jégou. Training data-efficient image transformers & distillation through attention. CoRR, abs/2012.12877 2020. URL wtt.aaivvv....///aa//..a//aaatvv org htt///////// 2012. 12877
David C. Van Essen, Stephen M. Smith, Deanna M. Barch, Timothy E.J. Behrens, Essa Yacoub, and Kamil Ugurbil. The WU-Minn Human Connectome Project: An overview. Neurolmage, 80: 62-79, 10 2013. ISSN 10538119.
Diego Vidaurre, Stephen M Smith, and Mark Woolrich. Brain network dynamics are hierarchi- cally organized in time. Proceedings of the National Academy of Sciences, 114(48):201705) 2017.
Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking, 2023.
Haiguang Wen, Junxing Shi, Yizhen Zhang, Kun-Han Lu, Jiayue Cao, and Zhongming Liu. Neural encoding and decoding with deep learning for dynamic natural vision. Cerebral cortex, 28(12): 4136-4160, 2018
James C Whittington, Joseph Warren, and Timothy E J Behrens. Relating transformers to models and neural representations of the hippocampal formation. December 2021.
Daniel LK Yamins and James J DiCarlo. Using goal-driven deep learning models to understand sensory cortex. Nature neuroscience, 19(3)355-335,, 2016.
BT Thomas Yeo, Mert R Sabuncu, Tom Vercauteren, Nicholas Ayache, Bruce Fischl, and Polina Golland. Spherical demons: fast diffeomorphic landmark-free surface registration. IEEE trans- actions on medical imaging, (9(3):650-663 2009.
BT Thomas Yeo. Fenna M Krienen, Jorge Sepulcre, Mert R Sabuncu, Danial Lashkari, Marisa Hollinshead, Joshua L Roffman, Jordan W Smoller, Lilla Zöllei, Jonathan R Polimeni, Bruce Fischl, Hesheng Liu, and Randy L Buckner. The organization of the human cerebral cortex estimated by intrinsic functional connectivity. Journal of Neurophysiology, (06(3):1225-116) 2011. doi: 10. 003.1////..0.3...... 201 1. Epub 2011 Jun 8.
15

---

Published as a conference paper at ICLR 2025
A APPENDIX DATA PROCESSING
A.1 7T TASK-FMRI HCP DATA
Functional MRI data downloaded from the Movie Task fMRI 1.6mm/59kk FIX-Denoised pack- age available for download at lattps://dk humanconnect ome org/ Cifti files were sep- arated into left and right hemispheres; then resampled from native resolution (59292 vertices) to Ja resolution (40962 vertices). This resampling is necessary to integrate with the SiT framework, which utilises regular icosahedral grids (c.g. 13) patch the input surface data (at lg).
A.2
VIDEO & AUDIO DATA PROCESSING
All IMRI sessions (MOVIEI-4) were divided into non-overlapping 3s .mp4y movie-clips ing opency Audio files were extracted to waav format at 16kHz from all movie clips with Eorchaudio library.
Choice temporal length of stimuli The duration 3s for movie clips was chosen for two reasons. First, it corresoonds to the average 'movic-shot" duration for most movies shown during the different fMRI sesions as detailed in Table 2 Second, a three-frame reconstruction yielded the best results during the vsMAE pre-training, which motivated the use of 3s movie clips (as TR 1). Qualitative and quantitative assessments frame reconstruction are provided in Appendix C.7
Table Average 'movie-shot" duration (as defined in PySceneDetect) for each movie in each fMRI session. We used PySceneDetect, to extract discting 'shots" from each movie. PySceneDetect is standard scene detection library with simple algorithm based on changes between rolling window of frames in the colour space to find fast cuts (the rolling window is to avoid considering fast camera movements as a scene cut). We verified visually that the output scenes were coherent.
HCP 7T EMRI session
Movie sample ID
Average 'movie-shot" duration
MOVIEI_CO
3.82
MOVIEI
2
1.71
MOVIEI CCI
3
3.0
MOVIE1 CC1
4
2.42
MOVIE2 HOI
4.28
MOVIE2 HO1
2.94
MOVIE2 HO1
3
3.56
MOVIE3 CC2
2.98
MOVIE3 CC2
2.80
MOVIE3_COT
3
2.25
MOVIE3.CO
3.77
MOVIE4 HO2
7.03
MOVIE4_HO2
2
6.73
MOVIE4_HOF
3
4.89
B APPENDIX METHODS
B.1 VSMAE PRE-TRAINING ARCHITECTURE DETAILS
In the Table B.2 we summarise the vsMAE architecture used for pre-training the SiT encoder ($ (RR, on video-fram reconstruction. Both encoder (#/MRJ) and decoder MMR, RM, are based ene
on the SiT-small architectures (Dahan et al. 2022 A classification token [CLS] is appended and used alongside the token sequence during the pre- pre-training. is used for the visualisation of attention maps.
16

---

Published as a conference paper at ICLR 2025
Table B.I: All SiT models preserve hidden size of 64 per attention head. The entire vsMAF encoder-decoder pipeline has in total 32M parameters. Here, the parameter count includes the initial linear projection layer from patch resolution (45 vertices per patch) and fMRI frames (3 chan- nels) to the embedding dimension D, and the final projection layer to resample the sequence to Ig resolution.
Models
Layers
Heads
Hidden size
MLP size
Params.
BfMRI
12
6
384
768
21.3M
ene & MMRI dec
6
6
384
768
10.7M
B.2 VSMAE PRE-TRAINING TRAINING DETAILS
Here, we summarise some essential training details for the vsMAE pre-training task.
While the temporal redundancy pixels in natural videos, allows for effective agnostic masking with high masking ratios (of up to 90%) (Feichtenhofer et al. 2022 He et al.)
Masking ratio
202T,
iMRI activations are less structured and much more noisy. Following results in Appendix masking ratio of p 50% in all vsMAE pre-training.
C.7
Masking strategy Similarly, the lack of structure and low temporal resolution in the spatio- temporal dynamics of brain activity is not suitable for agnostic masking strategy as (Feichtenhofer et al. 2022 Therefore, here employ sube-masking strategy as in Tong al. (2022) Wang al.
2023 where consecutive frames are masked with the same mask.
Temporal modelling Comparatively to Dahan et al. (204)). here we do not concatenate the se- quences of tokens from successive frames into single token sequence; however, concatenate fMRI patches from successive frames along the channel dimension. This design choice primarily due to the complexity of the self-attention operation with long sequences.
B.3 STIMULI PROCESSING AND ENCODINGS
For each 3s non-overlapping movie-clip, video-latent feature representations were extracted from pre-trained VidcoMAE model (Figure E) (trained on the Kinetics dataset for video understanding (Kay et al. 2017)). Matched audio representations [ were extracted using the wav2vec2.0
model (Baevski et al. 2020) pre-trained on 960 hours of unlabeled audio from LibriSpeech (Panay- otov et al. 2015) and available via torchaudio (torchaudio, wwppplliiiieee
WAV2VEC2 ASR_
BASE_9 960)). BASE_960H). Video and audio latent representations correspond to the token sequences extracted via PyTorch hooks from the penultimate layer each network (i.e. prior to the classification head). In both cases pre- pre-trained video and audio models were kept frozen during all trainings and experi- ments. Code for scene extraction can found in: PySceneDetect
B.4 FRAMEWORK ARCHITECTURE DETAILS AND TRAINING TIME
In Table B.2 we report the number of parameters used by each component of the SIM pipeline.
Table B.2: Parameter counts for the different component of the SIM architecture.
Models
Number of Parameters
SiT (DMMRI
21.3M
Mapper S FMRI
0.27M
Mapper fg fuideo
0.44M
Mapper fo
faudio
0.44M
htttosss/Atthbb.cccomm/mmmmm///m
17

---

Published as a conference paper at ICLR 2025
The total training time for training the SIM pipeline (finetuning of the & and training of the multimodal mappers for tri-modal CLIP alignment) of 24 hours on a cluster of 4 NVIDIA V100 GPUs with 32GB memory (internal cluster). Preliminary experiments (for development and hyperparameter tuning) took around 10 days of computing time on that same cluster. The vsMAE self-supervised pre-training took total training time of 2 days.
C APPENDIX RESULTS
C.1 EXPERIMENTAL DESIGN
To further clarify the experimental design, we complement Figure 2 with an illustration of the mul- tilevel train/test sampling setup Figure C.I
same
new
subject
IMR sussion
movie
same new
movie scenes
movie clips
Figure C.I: Our multilevel sampling setup: the dashed lines highlight levels at which train and test time sampling diverges. For the remainder levels, no difference in is made train and test time.
C.2 EXTENSIVE RETRIEVAL RESULTS EXPERIMENTS 1,2 AND 3
Table C.1 is a more comprehensive version of Table with results for soft-negative and hard- negative sampling in Experiment 1, including sensitivity study to the training method (ablation study). Note, since the CLIP optimisation is performed symmetrically for any pairs of modalities, the evaluation pipeline can be similarly applied to encode fMRI activations from movie embeddings. We denote this evaluation as V + fARI and > fMRI; showing that we are equally well able to predict fMRI from video audio, in reverse (Table [ bottom 2 rows). Table C.1 C.I reports the results in four inference directions: [YRI - V, [RR A. A - [RRI and V MMRI Results show that for all inference directions, performance drastically improves when we pre-train the SiT the vsMAE self- self-supervision task. This highlights the contributions of
IMRI encoder ($ enc using surface reconstruction as a pre-training to learn spatio-temporal dynamics of brain activity. Fine-tuning further boosts top-1 performance for all models (and top- 10 performance for most).
Table
compile all the results for Experiment 2 and Experiment presented in Figure
and
gure
18

---

Published as a conference paper at ICLR 2025
Table C.I: Extensive results for the Experiment 1. We report top-1 and top-10 retrieval results when generalising to new subjects. Bold corresponds to the best-performing model for each infer- ence modality. We compare random, ridge regression baseline and SiT encoder (&/MR) with our proposed tri-modal CLIP loss. We add here different combinations of training modalities, use of vs- MAE pre-training (versus training from scratch), and fine-tuning. training mode refers (1) frozen (training only the multimodal mappers (with 96% decrease in parameter count); (2) training from scratch; (3) finetuning from pre-trained weights. Results (in %) with jr and 95% conf. interval (CI).
Inference Modality
Training modalities
Training soft-negative sampling hard-negative sampling mode
Methods
Pre-traie Pre-training
top-1
top-10
top-1
top-10
Random
X
2.0+0.33
17.5=0.6
3.7
19,4+10.9
Ridge fynu, V
X
(2)
10.8+00.7
44.5+1.4
15.6 +1.1
64.1 11.6
SiT
fyRU, V
vsMAE
(I)
67.7+2.3
92.4+1.7
57.6+2.2
94.4+11.5
SIT
fARI, V
(2)
39.2+2255
87.7+2.6
50.2+2.55
1.75+1177
JARI +
SIT
fURI, V
vsMAE
(3)
79.5+2.4
94,8+1.5
64.7 2.3
95.6+1.44
SiT
fAIRI, V, A
vsMAE
(I)
42.0+1.7
83,2+1.9
37.6+1.7
89.7
SIT
hARI, V. A
X
56.6112.3
80.9
46.2.12.1
8.
SIT
FARI, V.A
vsMAE
80.3
92.1
76.8
94.2
Random
x FRRI, A
X
2.920.4
30.9#1.2
3.1=00.4
30.1
Ridge
(2)
3.6++0.5
1.1
3.2+0..4
1.2
fARI + A
SiT
fXRI, A fAnI, V,
vsMAF
(3)
2..1+
6..1+1.7
19.9+1.3
1.75+1177
SiT
vsMAF
70.9+
87.9+119
56.6
86.9++2.0
V fARI
SIT
fARI, V.A
vsMAE
(3)
74.8112.9
90.712.11
52.512.5
1.
A
fARI
SIT
KARI, V,A
vsMAE
(3)
61.1122.8
88.012.0
50.6:12.6
85.0.1..1
Table C.2: Extensive results for the Experiment Generalisation new movie scenes for all sub- jects. Retrieval accuracy on new stimuli of training subjects. M = 32 is used to account for the size of left-out movie scenes, Results (in %) with प and 95% conf. interval (CI). The hard-negative
results are shown in Figure C.3 and soft-negative results are shown in Figure C.2
Training
soft-negative sampling
hard-negative sampling
Methods
Modalitics
top-1
top-5
top-10
top-1
top-5
top-10
Random
X
3.6.10.2
17.910.4
33.2.1..5
4.6.10.3
20.4-0.5
39.8110,6
Ridge
fMRI, V
4.6.1 0.3
19.110.5
35.11106
7.110.4
30.0-0.7
58.2.10.8
SiT
fMRI, V,A
19.6 0.7
53.8
0.9
79.1 0.7
13.9+0.5
43.3=0.8
70.2+10.7
Table C.3: Extensive results for the Experiment 3: Generalization to new scenes and new subjects.
Retrieval Accuracy for stimuli of testing subjects. M 32 for inter-movie sampling but
M- = 16 for intra-movie sampling due to the limited size of left-out movie scenes. Results (in
%) with j and 95% conf. interval (CI). The hurd-negative results are shown in Figure C.3 and
soft-negative results are shown in Figure C.2
Methods Training Modalities
soft-negative
sampling
negatveg
sampling
top-1
top-5
top-10
top-1
top-5
top-
Random
X
4.5+0.4
17.4=0.7
30.510.8
5.5
21.110.7
40.0-10.8
Ridge fA V
5.2+:0.4
20.11008
36.21009
9.641..7
35.2+1.2
6..9+1.2
SIT
fMRI, V
19.2::1.1
50.1+:1.
79.21:11
12.810.9
43.611.2
80.7+1..1
C.3 ATTENTION MAPS COMPARISON WITH FUNCTIONAL NETWORKS
In Figure C.4
we compare the attention maps presented in Figure against the three first gradients from Margulies al. (2016) and the 7 resting-state networks from Yeo al. 201 1011).
19

---

Published as a conference paper at ICLR 2025
Experiment 2
Experiment 3
80
Model
Random
Riáge
S/T (ouns)
60
Accrrry
40
20
Topt Top0
Toplo
Top1
Tops
ToplD
Figure C.2: Soft-negative fMRI + V retrieval results for Experiment 2 and 3. Negative pairs (31) are sampled from different movies than the positive sample - but sampled from new movie scenes (not used during training) showing generalisation to new stimuli for train subjects (Experimeni 2) and new subjects (Experiment 3), as detailed Figure 2 Results (in %) with ji and 95% conf. interval. Two-sample t-tests (Ridge VS SiT) with Bonferroni correction were highly significant < 0.001)
Experiment
Experiment
80
Model
Random
Rodge
SIT (ours)
I
Aomracy
I
I
Top1
Top5
Top10
Top1
Top5
To910
Figure C.3: Hard-negative fMRI + V retrieval results for Experiment 2 and Experiment 3. Due to the experimental setup of Experiment the number of samples is limited for hard-negative sampling. Therefore, here results are present with M 32 for Experiment 2 but M = 16 for Experiment 3. Results (in %) with f and 95% conf. interval. Two-sample t-tests (Ridge VS SiT) with Bonferroni correction were highly significant (p 0.001)
C.4 ATTENTION MAPS ANALYSIS
Using the 50 topics from the Neurosynth maps, correlated the attention maps shown in Figure 4 highlight the brain function associated with each attention heads. We show the result in Figure C.6 Topic can be found This results confirm the specialisation of each attention heads to extract signal from specific func- tional networks.
org/ analyses/aaaaaaaa httppics//-ttopiccaes
20

---

Published as a conference paper at ICLR 2025
Attention Map
Artention Map
Artention Map
Yoa RSNs
Allention Map
Atlention Map
Somatomotoo
Gradent
Gradent
Gradient
Dorsi
Venta Artentin
Limbic Nowork
Fromooprieea
Figure C.4: Comparison between the attention maps extracted as in Figure gradients maps and Yeo et al. (201I) functional networks
Margulies et al. (2016)
All menronn
(b)
94
-01
02
No Sacvert
Figure C.5; Pearson correlation between Margulies et al. (2016) gradient maps, Yeo et al. (001)
functional networks and the six average attention heads as described Figure
C.5 RETRIEVAL RESULTS
Additional decoding results are provided, while retrieving from [ARI - V. Figures C.7 and Figure C.8 show retrieval results, respectively, for intra-movie sampling and inter-movic sampling for un- seen subjects (trained on the same stimuli). These retrieval results provide additional insights into the generalisation capabilities of the tri-modal clip alignment for stimuli retrieval in large pop- ulation. Figure C.9 provides additional results for stimuli retrieval in wnseen subjects and unseen stümuli.
Figure C.8 and Figure C.7 show visual examples for decoding performance when inferring fMRI V. This shows that not only is the selected top-1 clip often correct but also that all frames in the top 3 show semantically similar information. By contrast the bottom 3 clips are clearly very different.
21

---

Published as a conference paper at ICLR 2025
Attention Head
Attention Head
Attention Head
par fereece expritive www.ssttttnnnnnooooooooooo centrol
language_ heageee_eddinnnniee
henisphere, Janguage_stroke visual_ cortox cenoorrrrrrrrrrrrrr
audi tary speecceechh.ccc temporal Botton_ wwwpppreetonotteeeeeoeoee
auditory setprrysppeeh_eeeppppeeee tesporal
action, titis observation
ann.ffornlljifeee infferie
action actions observation Jearning_traaninng
wwe.franal_inffei
notor cortex
retor_cortex_hne
semuttic_catggerynoo wtw..aattpory..rreentttteet
Searning trasting prectice
cortec cingulate
face faces facial pew.inperception.iiii
DYNAL Froccal Lifferaar el Len peelllteget
per.eentlll............. events
ATragellen,
visual_cortex col_ttteeeesssrry
face faces_ Facial
visual sortte__sensoo
semaniic Crrcgoy pttpprr..pppproooeeeppe Epessaataioosss
auditory
notion_ Doctionperreetion.Vee visual
spatial body,
exgretic mechentine
attention, attention_attentionn) atteentionnl_targett
Attention Head
Attention Head
Attention Head
Figure C.6: Neurosynth topic correlation with Attention maps shown Figure
22

---

Published as a conference paper at ICLR 2025
Reference video clip
top
tap2
top3
top30
top31
top32
Test subject
Tesc: subject 2
Test subject
Text subjject
Figure C.7: hard-negative sampling results for new subjects. The model can generalise stimuli- retrieval for new subjects (trained on the same stimuli). The top-ranked movie-clips semantically close.
23

---

Published as a conference paper at ICLR 2025
Reference video clip
topl
top2
top3
top62
top63
top64
Test subject
Test subject
REST
Test subject 3
Test subject
Test subject
FEST
Figure C.8: sofl-negative sampling results for new subjects. Here, the model can correctly rank stimuli among a large pool of movie-clip samples (64), clustering similar concepts together from different movies: nature (subject test and 4), people talking (subject test and 5), kids (subject 3).
24

---

Published as a conference paper at ICLR 2025
Figure C.9: Video retrieval for unseen scenes and wiseen subjects. (a) corresponds inter-mover sampling: top-ranked movie clips are all dialogue scenes; (b) corresponds to intra-mvvic sampling: top-ranked video clips all show human and bright-color stimuli.
Referonne
topés
tap23
tmpéo
tapé
subject
Reforoncc vidoo
top1
top2
top8
70p4
top6
top28
10p20
10p30
10p31
Top82
(9)
C.6 RECONSTRUCTION RESULTS
In Figure C.10 and Figure C.10 we provide additional video-frame reconstruction results from the Experiment Figure C.10 shows the variability in the reconstruction results between different test participants, while C.II shows the diversity in semantic reconstruction. Overall, our SIM model can generalise to new subjects without having to fine-tune or train model on test subjects.
(a) Experiment 1 new subjects
Impur Frames
Teee Subject
Reconstructed Frrnmes
Test Subjectt
Test Subject 3
Figure 10 Video-frame reconstruction results from fMRI embeddings after tri tri-modal CLIP align- ment. Here, show the reconstruction results for 3 test subjects for the same movie-frame, demon- strating that the SIM framework can generalise to individual brain activation.
25

---

Published as a conference paper at ICLR 2025
Inpat
Raconattrra Freumis
Figure C.11: Video-frame reconstruction results from fMRI embeddings after tri-modal CLIP align- ment. Here, we show various Video-framt reconstructions for one test subject, showing that the SIM framework captures various semanties.
C.7 VМA PRE-TRAINING MASKING RATIO
The masking ratio parameter is crucial hyperparameter in the vsMAE pre-training task as it impacts both reconstruction quality and training time. We evaluated the performance various masking ratios (25%, 50%, 75% and 90%), frame reconstructions for the task fMRI HCP data. In Table C.4 we report the average MSE video-frame reconstruction error over 3 training runs, for different masking ratios p [25%, 50%, 90%].
Masking Ratio p MSE Reconstruction Error
25%
0.78 + 0.05
50%
0.39 + 0.03
75%
0.49 0.03
90%
0.68 = 0.05
Table C.4: Reconstruction errors (MSE) are reported on the validation set for the 3-frame recon- struction task. MSE is calculated only for the masked tokens. In all cases, vsMAE self-supervision was done for 50,000 iterations with a batch size of 64. Validation errors with stds across three runs are reported.
Qualitatively, we report reconstruction results Figure C.12 The reconstruction quality is high, capturing dynamics of functional connectivity over successive frames, with high-masking
tios.
Overall, the 50% masking ratio offered the best quantitative validation and was used in all periments. Such observations are aligned with those in He et al. (2021), where it was noted that masking high percentage of patches is necessary reduce redundancy and create challeng- ing self-supervisory task which leads to learning more meaningful weights; however, increasing the masking ratio above certain threshold leads higher reconstruction errors and consequently higher prediction errors.
C.8 TEMPORAL LAG
The choice of temporal lag of T = 6 was validated by calculating the correlation between the video latent embeddings (Xv) (extracted from the pre-trained videoMAE (Tong et al.) 2022) and 3s movie-clips) and the corresponding 3s delayed by various temporal lags 7 E 3,,6,6 10), following methodology in Huth et al. (2016). This was achieved by training a ridge regression model for each HCP participant, on compiled MOVIEI-3 fMRI sessions and predicting the individual timeseries for the missing fMRI session (MOVIE4). An example of the map after statistical correlation Wilcoxon and Bonferroni correction, shows vertices with p-value< 10e in Figure C.15 lag of
By using temporal
6, the correlation values were the highest.
26

---

Published as a conference paper at ICLR 2025
Ground
Recondrr
50%
abbst
Fame 2
WMAE (raee))
Fame
Test sarapst 2
Trams
Figure C.12: fMRI reconstructions result from vsMAE pre-training on 3-frames reconstruction and trained with masking ratio م 50%. Results for two test subjects presented, showing high- quality reconstruction of individual brain activation.
masking ratio 50%
Truth
Input (Maked)
on
Frame
Frime
Text carject
Frame
Frame 10
Frams
Frame 16
Figure C.13: fMRI reconstructions result from vsMAE pre-training 16-frames reconstruction and trained with masking ratio p = 50%. Reconstruction results are presented for frames from 16s time window. The vsMAE effectively captures spatio-temporal dynamics of brain activity while preserving individual features.
27

---

Published as a conference paper at ICLR 2025
Ground Truth
Input (Masked)
Resonstructon
75%
Frame 12
Testsupett
Framo
Frume 36
Frame 48
Frimo 60
Figure C.14: MMRI reconstructions result from vsMAE pre-training on 64-frames reconstruction and trained with masking ratio p = 75%. Reconstruction results are presented for 6 frames from 64s time window. The vsMAE captures global patterns of brain activity in the form of functional networks.
10
-1.0
0.3
Figure C.15: Correlation map of predicted fMRI timeseries from video latent representations for the IMRI session MOVIE4. with temporal lag of These maps are averaged across test subjects and statistically corrected for Wilcoxon and Bonferroni correction. Most of the visual and auditory systems are highly correlated.
28


Reviews:
[{"review_id": "lAcqB40M9I", "reviewer": "Reviewer_zoi6", "summary": "This paper proposes a new technique for decoding brain signals that is generalizable across participants. They integrate surface vision transformers (SiTs) with multi-modal contrastive learning to decode audio and visual stimuli from cortical fMRI data. They demonstrate the efficacy of their approach on the HCP movie watching database and show superior decoding performance. The model shows strong performance not only on the same subjects and stimuli seen during training but also on new subjects and previously unseen stimuli", "questions": "- During inference, why not use the full test set for retrieval? The size of the set clearly has an impact on the retrieval performance. Why do the authors choose a lower set size for the auditory domain? \n- Can the authors comment on the biological plausibility of the encoder architectures used for stimulus encoding - eg wav2vec which works directly with raw audio waveforms instead of cochleograms/spectrograms may not be the best choice for reformatting auditory signals into brain-aligned representations", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The idea of multimodal alignment of vision, audition and fMRI through contrastive learning seems novel and promising \n- The study demonstrates that the model can generalize across unseen subjects and stimuli, which is a crucial aspect of developing robust brain decoding methods.\n- The paper builds on established techniques in brain decoding by introducing more advanced architectures, like surface vision transformers (SiTs) which are better able to capture long-range structure in surface imaging data", "weaknesses": "- Comparison against alternative architectures like 3D CNNs etc is lacking in this work, which makes it harder to understand the contribution of the Surface vision transformer \n-  Quantitative evaluations of the reconstructed visual clips against other state of the art decoding models is also currently lacking in those work. At the very least, the authors should include more examples of reconstructed clips\n- Model performance is presented using retrieval tests which are much easier than reconstructions; Given that most studies nowadays can perform decent decoding with the help of powerful generative models, this should atleast be discussed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "n5MDURlUeE", "reviewer": "Reviewer_sizD", "summary": "The authors present a novel approach to modeling cortical functional dynamics by leveraging Vision Transformers (ViTs) tailored for surface data analysis. Their method integrates a tri-modal alignment using CLIP, aligning audio, video, and fMRI data to enhance the retrieval of visual and auditory stimuli from cortical activity patterns and vice versa. The model demonstrates high within-subject accuracy, effectively identifying audio and video samples from just 3-second fMRI recordings. Additionally, the authors investigate attention maps that reveal distinct patterns associated with the brain's semantic and visual systems. They explore model's generalizability on (i) unseen subjects; (ii) unseen stimuli; and (iii) unseen stimuli within unseen subjects.", "questions": "What constraints or shortcomings resulted from training the models using only 3-second temporal windows? How might this impact the model's ability to capture longer-term temporal dynamics or more complex cognitive processes?\n\nThe caption for Figure 4 lacks clarity in identifying specific brain regions. It would be more informative to directly label or annotate the relevant areas (such as Broca's area, auditory cortex, sensorimotor cortex, visual areas, and the face area) on the attention map visualizations.\n\nHow does performance compare to other state-of-the-art brain decoding methods?\n\nWhat are the privacy/ethical implications of being able to reconstruct visual experiences from brain data?\n\nThe study uses a fixed temporal lag of 6 seconds to account for the hemodynamic response. However, the hemodynamic response lag is known to vary across different brain regions. How might this fixed lag approach impact the model's ability to accurately capture temporal dynamics in regions with differing hemodynamic response functions?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The authors: \n- design a novel framework composed of SiTs and tri-modal CLIP alignment for brain decoding. \n- evaluate the model on unseen subjects and unseen stimuli\n- leverage attention maps providing interpretability of the model's behavior, and make the connection between the information extracted from attention heads and specific functional networks\n- show the recontruction of visual stimuli improves through the tri-modal approach", "weaknesses": "- Limited to decoding 3-second clips, may not capture longer-term temporal dynamics\n- Only tested on movie watching data, unclear how well it would generalize to other tasks/stimuli\n- Requires high computational resources (multiple A100 GPUs)\n- Limited diversity in the movie stimuli used for training", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "434L4AX4Zp", "reviewer": "Reviewer_vWQL", "summary": "The paper addresses the challenge of generalizing decoding models for video and audio stimuli to fMRI responses from subjects not included in the training data, aiming for a universal cognitive model rather than subject-specific ones typically achieved in this field. The authors propose an approach that combines anatomical and functional alignment to map across subject-specific response spaces. Their method leverages two main principles: topographical alignment and space-time pattern matching. To implement this, they use a Vision Transformer with cortical surface response patches as input, alongside a tri-modal CLIP criterion imposed on a shared embedding space for audio, video, and fMRI. The approach is evaluated on the 7T HCP movie dataset with over 170 subjects, demonstrating successful nearest-neighbor identification of video and audio samples from responses of held-out subjects. Additionally, some success is shown in the fully zero-shot setting, where both the test subject and test stimuli are unseen during training.", "questions": "1) I suggest creating figures with plots to visually represent the data. Each figure should have clear takeaways that are explicitly stated in their captions to guide the reader's understanding.\n\n2) Why is the inter-movie accuracy typically higher than the intra-movie accuracy in Tables 1, 2, and 3? Given that inter-movie represents the held-out movie case, which is expected to be more challenging, this finding is counterintuitive and warrants further explanation.\n\n3) It would be beneficial to apply the method to another dataset, ideally focusing on a single modality. This would help demonstrate the robustness and generalizability of the approach.\n\n4) If feasible, I recommend breaking down the impact of the Vision Transformer (ViT) compared to the CLIP approach. This could involve substituting CLIP with a more conventional training and classification method (e.g., referencing works from the Kamitani and Gallant group over the past decade). Such a comparison would strengthen the case for the specific contributions related to subject canonical space mapping, rather than merely showcasing advanced training and classification techniques.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Significant Challenge Addressed: The paper tackles the important and well-recognized challenge of inter-subject generalization in brain decoding tasks. By striving for a universal cognitive model that can apply to responses from unseen subjects, the authors make a meaningful contribution to the field, which often relies on subject-specific models that limit broader applicability.\n\n- Innovative Methodology: The authors present an intriguing combination of a Vision Transformer (ViT) applied to patches of cortical surface responses, paired with a multi-modal contrastive learning criterion. This approach leverages the strengths of ViT for spatial representation while effectively integrating different modalities (audio, video, fMRI) to enhance the decoding process.", "weaknesses": "The main issue is the combination of too many ideas within a single paper. While the focus on inter-subject generalization in stimulus decoding is critical, the inclusion of multi-modal audio and video adds complexity that obscures the primary contributions. The rationale for combining these modalities should be clearly justified, particularly since generalization across subjects poses challenges even in simpler stimulus domains like images. A more focused exploration of the decoding problem could strengthen the paper's impact.\n\nThe results are difficult to interpret, especially in Table 1. The meaning of the \"inter\" and \"intra movie\" columns needs clarification. It would be beneficial to restrict the table to test movies included in the training dataset. Additionally, an evaluation of audio versus video identification success is lacking. The absence of same-subject (intra-subject) baseline results and chance-level metrics makes it challenging to assess the significance of the reported accuracies (e.g., 80% vs. 77% top-1 accuracy). The authors should enhance guidance in the text to help the reader grasp the key contrasts of interest.\n\nFurthermore, the decoding problem addressed appears to be a nearest-neighbor classification, which is arguably a basic form of decoding. Given the recent advancements in within-subject stimulus reconstruction, a more compelling approach would explore inter-subject generalization in the context of these more popular and challenging brain decoding tasks.\n\nAdditional comments:\n- The originality of the work is challenging to evaluate. The authors should more thoroughly situate their contributions within the context of previous research, particularly in relation to cited works by Dahan et al. (2024), Tong et al. (2022), and Feichtenhofer et al. (2022), emphasizing the added value of their approach.\n- The paper lacks plots that could effectively convey findings in a more visual and intelligible manner. Incorporating proper uncertainty estimates in graphical representations would enhance clarity.\n- The writing style is overly cryptic, with excessive notation making the paper difficult to follow. Simplifying the presentation could greatly improve readability.\n- The figures, particularly in the main figure (Fig. 1), suffer from poor quality; the font sizes are too small, and the caption is not sufficiently explanatory.\n- There are numerous grammatical and syntactical errors throughout the manuscript. A thorough review is necessary to enhance clarity and professionalism.\n- The authors only consider a single dataset. Demonstrating the applicability of their approach across multiple publicly available datasets would strengthen the paper’s generalizability.\n- The authors use the terms \"movie\" and \"video\" interchangeably, creating ambiguity regarding whether \"movie\" refers to video with audio or just video alone. This should be clearly defined.\n- Main results, such as those in Table 1, should not be conflated with ablation studies to avoid confusion.\n- The manuscript is missing comparisons to other identification/classification approaches, both within and across subjects, which would provide valuable context and relevance to the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tD5xwKQa73", "reviewer": "Reviewer_6xCn", "summary": "This paper presents a novel approach for brain decoding using surface vision transformers (SiT) and tri-modal CLIP contrastive learning to generalize audiovisual stimuli decoding from fMRI data across different subjects and unseen stimuli. The model leverages the spatial topography of the brain’s cortical surface, combined with alignment between fMRI, audio, and video embeddings, achieving up to 80% top-1 accuracy in identifying stimuli across subjects in a movie-watching task. The approach is validated on the HCP 7T movie-watching dataset, showing strong generalization and offering interpretability through attention maps that highlight brain regions involved in audiovisual processing.", "questions": "1. In section 4.4, the authors propose the potential for training encoding/decoding frameworks that exhibit generalizability. While the model shows robust generalization performance on the HCP 7T movie-watching dataset, have the authors considered evaluating its generalizability under more diverse experimental conditions, such as with abstract visual stimuli, silent films, or non-narrative content? Given that the HCP dataset primarily includes specific cinematic types, how do the authors anticipate the model’s performance on stimuli with varying visual and auditory properties?\n\n\n2. In Table 1:\n-  The vsMAE pre-training appears to significantly enhance performance compared to training from scratch. For instance, in the case of $f_{MRI}\\rightarrow \\mathcal{V}$ with SiT, vsMAE pre-training substantially improves the top-1 metrics. Could the authors elaborate on how this pre-training enhances the model’s generalization capabilities? Is this improvement consistent across different stimulus types (e.g., non-movie visual content), or is it specifically tied to the training dataset?\n\n   - Some results exhibit considerable variance. For example, the results for $f_{MRI} \\rightarrow \\mathcal{A}$ with SiT show $70.9 \\pm 20.3$, and $f_{MRI} \\rightarrow \\mathcal{V}$ shows $80.3 \\pm 18.0$. Could the authors provide an explanation for this variability?\n\n3. I am a little bit confused about the information that Figure 4 wants to present. Could the authors clarify how these maps are quantitatively validated against known functional networks (Yeo et al), and how confident we can be that these maps are genuinely reflecting underlying neural processes rather than model artifacts?\n\n4. The overall generalizability of the model was evaluated using the same dataset. From my understanding, the contribution to generalizability here is that the model improves upon previous methods (which focused on single-subject analysis) by making it more generalizable to future tasks, but still within the same study population or dataset. Is this correct? Could the authors comment on this? Typically, when referring to generalizability, it implies extending the model’s applicability to a broader population or more diverse datasets. \n\n5. In the discussion section, the author mentioned that the current model does not include sub-cortical structures, despite their known role in sensory processing, such as the Lateral Geniculate Nucleus (LGN) in vision. Is it viable to handle the noise in subcortical area using CIFTI dataset?\n\n6. The authors utilize sine-cosine positional embeddings to encode the locations of cortical patches in the SiT model. Have the authors evaluated the effectiveness of this choice for representing the complex geometry of the brain’s surface? Would alternative methods, such as learned positional embeddings, improve performance or better capture spatial relationships in cortical data?\n\n7. What's the computation time?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The use of tri-modal CLIP alignment for fMRI, audio, and video embeddings is an innovative contribution, extending prior unimodal or bimodal approaches.\n\n- The integration of state-of-the-art techniques, like vision transformers and contrastive learning, is well-grounded and technically solid. Also, the model is rigorously evaluated on unseen subjects and stimuli, demonstrating strong performance in terms of Top-K accuracy.\n\n- Its potential impact on brain-computer interfaces (BCIs) and neuroscience research is significant, with promising future applications.", "weaknesses": "**Limited Generalization Beyond the Dataset**: The model’s generalization is limited to the HCP 7T dataset, with no evaluation on more diverse or real-world stimuli.\n\n**Computation**: the model’s self-attention mechanisms are computationally expensive, limiting scalability to higher resolutions.\n\n\nSome of the typos that need to be corrected:\n\n1. Appendix C.1, under Table C.1.'s caption. \"3-frame reconstruction taskon\", should be \"3-frame reconstruction task on\"?\n2. Line 512, \"evalute\" should be \"evaluate\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)