Sample-efficient decoding of visual stimuli from fMRI through inter-individual functional alignment
Authors: Alexis Thual
Yohann Benchetrit
Felix Geilert
Jérémy Rapin
Iurii Makarov
Stanislas Dehaene
Bertrand Thirion
Hubert Banville
Jean-Remi King
Venue: Accepted by TMLR ()

Abstract: Deep learning is leading to major advances in the realm of brain decoding from functional Magnetic Resonance Imaging (fMRI). However, the large inter-individual variability in brain characteristics has constrained most studies to train models on one participant at a time. This limitation hampers the training of deep learning models, which typically requires very large datasets. Here, we propose to boost brain decoding of videos and static images across participants by aligning brain responses of training and left-out participants. Evaluated on a retrieval task, compared to the anatomically-aligned baseline, our method halves the median rank in out-of-subject setups. It also outperforms classical within-subject approaches when fewer than 100 minutes of data is available for the tested participant. Furthermore, we show that our alignment framework handles multiple subjects, which improves accuracy upon classical single-subject approaches. Finally, we show that this method aligns neural representations in accordance with brain anatomy. Overall, this study lays the foundations for leveraging extensive neuroimaging datasets and enhancing the decoding of individual brains when a limited amount of brain-imaging data is available.

Published in Transactions on Machine Learning Research (05/2025)
Sample-efficient decoding of visual stimuli from fMRI through inter-individual functional alignment
Alexis Thual
alezis, Memis.thull@inrrion
Cognitive Neuroimaging Unit, INSERM, CEA, CNRS, NewroSpin center, Gif sur Yuette, France Mind, Inria Paris-Saclay, Palaiseau, France Inserm, Collège de France, Paris, France
Yohann Benchetrit
Meta AI
Félix Geilert
Meta A1
Jérémy Rapin
Meta Al
Iurii Makarov
Meta Al
Stanislas Dehacne
Cognitive Neuroimaging Unit, INSERM, CEA, CNRS, NeuroSpin center, Gif sur Yuette, France Inserm, Collège de France, Paris, France
Bertrand Thirion
Mind, Inria Paris-Saclay, Palaiseau, France
Hubert Banville Meta AI
Rémi King
jeanremi@meta.cco
Meta
Laboratoire des systèmes perceptifs, École normale supéricure
PSL University
Reviewed on OpenReview: https: // openreveew. net/ forun? id= quJraN50DT
Work done while interning Meta

---

Published in Transactions on Machine Learning Research (05/2025)
Abstract
Deep learning is leading to major advances in the realm of brain decoding from functional Magnetic Resonance Imaging (fMRI). However, the large intter-nddvidual variability in brain characteristics has constrained most studies to train models on one participant at time. This limitation hampers the training of deep learning models, which typically requires very large datasets. Here, we propose to boost brain decoding of videos and static images across participants by aligning brain responses of training and left-out participants. Evaluated on a retrieval task, compared the auatomically-aligned baseline, our method halves the median rank in out-of-subject setups in low-data regimes. It also outperforms classical within-subject approaches when fewer than 100 minutes of data is available for the tested participant. Furthermore, we show that our alignment framework handles multiple subjects, which improves accuracy upon classical single-subject approaches. Finally, we show that this method aligns neural representations accordance with brain anatomy. Overall, this study lays the foundations for leveraging extensive neuroimaging datasets and enhancing the decoding of individual brains when a limited amount of brain-imaging data available.
B
Reference
time
subject
Bine
time
Brain features
Left-out
Alignment model
subject
lime
Low-level
decodier
Low-level encoder
Frame features
High-level
decoder
Eime
Movie frames
Low-leve level decoder
High-level encoder
Alignment model
High-level decoder
Left-out
Reference-lkke like brain features
Models trained the reference subject
subject
Figure 1: General outline of video decoding from BOLD fMRI signal in left-out participants A. For each frame associated with brain volume, one computes its low- and high-level latent representations using pre-trained encoders. Then, brain decoders (green) can then be fitted to map brain features onto each of these latent representations. B. The BOLD signal acquired from two participants watching the same movie can be used to derive an alignment model (purple) that maps voxels from the two participants based on functional similarity. C. Then, this alignment model can be used transform brain features of the left-out. participant into the brain features that match those of the reference participant. In particular, this allows one to decoders that have been trained a lot of data coming from reference participant, and apply them a left-out participant for whom less data was collected.
1 Introduction
Decoding brain activity The generative capabilities of deep learning have recently unlocked decoding mental representations from brain activity. Originally restricted to linear models (Mitchell et al.) 2004 Harrison & Tong 2009 Haynes & Rees 2006), the decoding of brain activity can now be carried out with deep learning techniques. particular, using functional Magnetic Resonance Imaging (fMRI) signals,
significant, progress has been made in the decoding of images (Ozcelik & VanRullen 2023: Chen al. 2023a
2

---

Published in Transactions on Machine Learning Research (05/2025)
Scotti et al.
2023
Takagi & Nishimoto, 2023; Gu et al.,
2023;
Ferrante et al.
2023
Mai & Zhang 2023
speech (Tagg et al.
2023). and videos Kupershmidt et al. 2022 Wen et al.
2018 Wang et al. 2022 Chen
et al. 2023b
Lahner et al. 2023 Phillips et al. 2022).
The bottleneck of inter-subject variability A core issue is that brain organization is highly variable across participants, which makes it challenging to train a single model on multiple participants using fMRI data. Therefore, with lew noteworthy exceptions (Haxby et al.) 2020; Ho et al. 2023 studies lypically time. With this constraint in mind, major effort has been put towards building IMRI datasets collecting a lot of data in limited number of participants (Allen) et al. 2022; Wen et al., 2017 LeBel et al. 2023 Pinho et al. 2018). Nonetheless, the necessity train and test models on single participant constitutes major impediment to using notoriously data-hungry deep learning approaches. In addition, generalization to new individuals is essential to the validation of discoveries.
train a brain decoder on a single participant
Functional alignment Several methods can align the functional organization - on top of the anatomy - of multiple brains, and thus offer a potential solution to inter-individual variability: differentiable warps of the cortical surface (Robinson al. 2014), rotations between brain voxels the functional space (Haxby et al., 2011), shared response models (Chen et al.) 2015 Richard et al. 2020), permutations of voxels minimizing
Thual et al. 2022), combinations of these approaches Feilong et al. 2022). More recently, several studies rely on deep learning models trained in a self-supervised fashion to build an embedding of brain activity, in hope that it could be meaningful across participants
optimal transport cost (Bazeille et al.] 2019
(Thomas et al. 2022 Chen et al. 2023a). However, to this day, it is not clear which of these methods offers the best performance and generalization capabilities [Bzeeille et al. 2021
Approach It is currently unknown whether any of the aforementioned methods improve the decoding of naturalistic stimuli such as videos, and how such hypothetical gain would vary with the amount of fMRI recording available in a given a participant. To address this issue, we leverage [MRI recordings of training participants to boost the decoding of videos and static images in a single left-out participant, as illustrated in Figure D This requires fitting two models: an alignment model and a brain decoder. The aligmment aims at making brain responses of a left-out participant most similar to those of reference participant. Here, we leverage optimal transport to compute this transformation using funational and anatomical data from both participants. The brain decoder which we will refer to as the decoder consists linear regression trained to predict the latent representations of movie frames or static images from the corresponding BOLD signals or beta, coefficients. We evaluate video and image decoding in different setups. In particular, we assess (1) whether decoders generalize to participants on which they not trained, (2) whether training decoder on data from multiple participants improves performance and (3) the extent to which functional alignment improves the aforementioned setups.
Contributions We first confirm the feasibility of decoding, from 3 Tesla (3T) fMRI, the semantics of videos watched by the participants (Wen et al. 2017p. We verify that this approach also performs well for the decoding of static images from 7T fMRI data (Allen et al. 2022). Our study makes three main novel contributions:
1. Compared to the baseline, functional alignment across participants boosts visual semantics decoding performance in left-out participants when the latter have a limited amount of data
2. Training a decoder multiple functionally aligned participants yields model with improved per- formance compared to training one model per participant, but anatomical alignment does not
3. The resulting alignments, computed from movie-watching data, are anatomically coherent
3

---

Published in Transactions on Machine Learning Research (05/2025)
2 Methods
Our goal is to decode visual stimuli seen by individuals from their brain activity. To this end, we train a linear model to predict latent representations shortened as latents of these visual stimuli from BOLD fMRI signals recorded in participants watching naturalistic videos.
In the data under study, brains are typically imaged at rate of one every 2 seconds. During this period, participant sees 60 video frames on average, or static image for the case of Allen et al. (0022 For simplicity, we consider the restricted problem of decoding only the first video frame seen by participants at each brain scan, Formally, regardless of the dataset, for given participant, let X E R be the BOLD response collected in v voxels over n brain scans and Y E RR", the m-dimensional latent representation of each selected video frame.
2.1 Decoding
Brain input There is a time lag between the moment a stimulus is played and the moment elicits a maximal BOLD response in the brain (Glover 1999). Moreover, the effect induced by this stimulus might span over multiple consccutive brain volumes. To account for these effects, we standard Finite Impulse Response (FIR) approach. It consists in fitting the decoder BOLD response. In particular, we refer to the number of brain volumes to aggregate together in the FIR approach as the window size. Different aggregation functions can be used, such as stacking or averaging. Figure $2 describes these concepts visually.
Lime-lagged, multi-volume version of the
Video output The matrix of latent features Y is obtained by using pre-trained image encoder on each video frame and concatenating all obtained vectors in Y. Similarly to Ozcelik & VanRullen (023)) and illustrated in Figure [ A, we seek to predict CLIP 257 x 768 (high-level) and VD-VAE (low-leve level) latent representations. We use visual - as opposed to textual - CLIP representations (Radford et al. 2021). For comparison, we reproduce our approach on latent representations from CLIP CLS (high-lovel) and AutoKL (low-level), which happen to much smaller P and are computationally easier to fit.
Model Fitting the decoder consists in deriving W E R b IR' the solution of Ridge regression problem i.e. linear regression with L2 regularization predicting Y from X. Note that u, the nunber of vertices, is the same for the brain decoder and for the brain alignment module.
We evaluate the performance of the decoder with retrieval metrics. Let us denote Xtrain and Yurain the brain and latent features used train the decoder, Xeet and Yest those to test the decoder, and Y W b the predicted latents. We ensure that the train and test data disjoint.
Evaluation
We randomly draw a retrieval set K of 499 frames without replacement from the test data. For each pair (g. y) of predicted and ground truth latents, one derives their cosine similarity score s(ğ y), as well as similarity scores to all latents Yneg of the retrieval set s( yneg). Let us denote (D, y) the rank of which we define as the number elements of K whose similarity score to ÿ is larger than s( y). In order for the rank to not depend on the size of K, we define the relative rank as r(ú. /) K). Finally, one derives the median relative rank MR(Y. Y K):
Fx T y) [yneg E K s(0, Yneg) > s(. y
MR Ý, K) median (rx((0. @median(1rx(ee 4) | K| (0.
2.2 Brain alignment
A nntooiccal alignment As baseline, consider the alignment method implemented in Freesurfer
Fischl 2012 which relies on anatomical information to project each participant onto surface template
Dimensions for CLIP CLS: 768
CLIP 257 x 768 257 768 197 AutaKL: 4 4096 VD-VAE:
24 +4x 28 210 212 + 214 168
4

---

Published in Transactions on Machine Learning Research (05/2025)
of the cortex (in case fsavemag55). Consequently, brain data from all participants lie mesh of size 142 vertices per hemisphere.
Functional alignment On top of the aforementioned anatomical alignment, we apply recent method Wasserstein (FUGW) As illustrated in Figure B, this mothod consists in using functional data to train an alignment that transforms brain responses a given left-out participant into the brain responses of a reference participant. This approach can be seen a soft permutation of voxels p of the left-out participant which maximizes the functional similarity to voxels of the reference participant.
from Thual et al. (2022) denoted as Fused Unbalanced
Formally, for left-out participant, let Dow .R. be the matrix of anatomical distances between vertices on the cortex, and w E R2 probability distribution on vertices. wout can be interpreted as the relative importance of vertices; without prior knowlodge, we use the uniform distribution. Reciprocally, we define D and wee for a reference participant. Note that, in the general case, v can be different from one participant to the other, although we simplify notations here.
We derive transport plan PE R to match the vertices of the two participants based on functional sim- ilarity, while preserving anatomical organisation. For this, simultancously optimize multiple constraints, formulated in the loss function Le (P) described in Equation
Gromov Wasserstein loss
Wasserstein loss
(e(P) A (1 a) Lw P) + Low(P)
(1)
+p Cu(P) tE H(P)
Marginal constraints
Regularization
Each component of the loss is expressed follows:
Lw(P) A Σ
Xout
X P.j
0Stj<C
Low(P)
Dout
D Pió Pa.
OSi,kjj
KT.
P+1 &
P#1 u N w=) + KT. & P#2lw" @ w²)
H(P) A KL & P
(w"
X w X (w" @ wt))
Here, KL(.-) denotes the Kullback-Leibler divergence, P#I A (E is the first marginal of P. P42 2 (E, Pra)osicn the second marginal P, a [0, P R_ are the hyper-parameterr setting the relative importance each constraint, and e - (Xout, X Dout Dree D p)
block coordinate descent algorithm (Séjourné et al.) 2021), each running 1.00 Sinkhorn iterations (Cuturi 2013). Subsequently, we define Pout-aref: X p XT) P#2 RR,. where & is the element-wise division, function which transports any matrix of brain features from the left-out participant to the reference participant. To simplify notations, for any X defined on the left-out participant, we define Yout->ref á dout ())
Following Thual et al. (2022). we minimize (e(P) with 10 iterations
Syper-paramctery selection for functional alignment We use default parameters shipped with ver- sion 0.1.0 of FUGW. Namely, a, which cons the balance between Wasserstein and Gromov-Wasse Wasserstein losses i.. how important functional data. is compared to anatomical data is set to 0.5. Empirically, we see that a = 0.5 yields values for the Wasserstein loss which are larger than that of the Gromov- Wasserstein loss, meaning that functional data drives these alignments. Secondly, p, which sets the importance of marginal
/tttps:/alexisthhaa fottuu.ioofuuuu We the mazel (volumetric pixel) or verter (point on mesh) indifferently.
5

---

Published in Transactions on Machine Learning Research (05/2025)
constraints i.c. to what extent more less mass be transported to from each voxel is set to 1. Empirically, this value leads to all voxels being transported / matched with equal importance. Finally, which controls for entropic regularization i.e. how blurry computed alignments will be is set to 10-4 Empirically, this value yiclds alignments which are anatomically very sharp, i.c source voxels are matched with a handful of larget voxels only (and versa..
3
Experimental setup
3.1 Decoding and alignment setups
Our main methodological contribution consists in training and evaluating brain decoders in variety of set in which participants have been functionally aligned or not.
Within- us out-of-subject Let us consider a decoder trained on data (X Igain: Y from given participant. The within-subject setup consists in testing it on left-out data (X same participant. The out-of-subjet setup consists in testing it on data (X participant.
ued: Y acquired in the test Yistt acquired in a left-out
Single- us multi-subject The single-subject setup consists in training decoder using data from one participant only. The mulli-subjecl setup consists in training a decoder using data from multiple participants. In this study, data from several participants are stacked, resulting in a matrix X E Rur+httpp and Ymulti E tp..t..t..,. where p is the number of participants.
Aligned us un-aligned Let S be the reference participant. In the out-of-subjoct and multi-subjoct setups, data coming from different participants can be functionally aligned or not that of the reference participant. It modifies these respective setups as follows: (1) in the out-of-subject case, it corresponds aligning S2 onto Sn, such that decoder trained on S will be tested on X S2 Y (2) in the multi-subject case, all participants are aligned to S1 and the decoder is trained
concatenation of S S S, (see notations introduced at the end of section 2.2 and ys,,.. YS,,
Setups of interest are visually described in Figure A.
Evaluation under different data regimes Note that the aligmment and decoding models do not need to be fitted using the same amount of data. In particular, we are interested in evaluating out-of-subject performance in setups where a lot of data is available for the reference participant, and little data is available for the left-out participant: this would typically be the case in clinical setups where, usually, little data is available in patients. In this case, we evaluate whether is possible to use this small amount of data align the left-out participant onto the reference participant, and have the left-out participant benefit, from decoder previously trained on lot of data.
3.2 Datasets
We analyze two MMR datasets. The first dataset (Wen et al. 2017 comprises 3 human participants who watched 688 minutes of video. The videos consists of train segments mimutes each and test segments of minutes each. Each training segment was presented twice. Fach test segment. was presented 10 times. Each segment consists of a sequence of roughly 10-second video clips. The fMRI data was acquired 3T, 3.5mm isotropic spatial resolution and 2-second temporal resolution. It minimally pre-processed with the pre- processing pipeline as that of the Human Connectome Project (Glasser et al. 2013). In particular, data from cach participant are projected onto common volumctric anatomical template. Similarly to prior work on this dataset (Wen et al. 2018 Kupershmidt el al. 2022 Wang el. al. 2022). we use runs related to the first video segments- 288 minutes as training data, and runs related to the last video segments as test data. Thus, it amounts 8640 training samples and 1200 test samples per individual.
The second dataset (Allen et al. 2021) denoted as the Natural Scenes Dataset (NSD) comprises 8
participants who shown 10 000 static images three times. They were scanned over 40 sessions of 60

---

Published in Transactions on Machine Learning Research (05/2025)
minutes, amounting to 2 400 minutes of data. This amounts to 72 000 samples per individual. Instead of raw BOLD signal, we leverage precomputed per-trial regression coefficients accessible online. See supplementary section A.6 for more details.
3.3 Preprocessing
For the Wen al. (017)) dataset, we implement minimal additional preprocessing steps for each participant separately. For this, (1) project all volumetric data onlo the FreeSurfer average surface template ssav- erage5 (Fischl 2012), then (2) regress out cosine drifts in each vertex and each run and finally (3) center and scale each vertex time-course in each run. Figure [ST gives a visual explanation as to why the last two sleps are needed. The first two steps are implemented with Nilearn (Abraham et al., 2014 Q and the last one with Scikit-Learn (Pedregosa et al., 2011)).
Additionally, for given participant, same video averaged, and a second one where they are stacked.
try two different setups: a first one where runs showing the
The Allen et al. (2021) dataset is already preprocessed by the original authors, and maps of beta coelficients from a General Linear Model are accessible online.
3.4 Hyper-parameters selection for decoders
To train decoders, use the same regularization coefficient Oridge across latent types and choose it by running cross-validated grid search folds of the training data. We find that results are robust to using different values and therefore set Aridge = 50 000. Similarly, values for lag, window size and aggregation function are determined through a cross-validated grid search. The loss used to determine these parameters in the relative median rank introduced 2.1
4 Results
4.1 Within-subject prediction of visual representations from BOLD signal and retrieval of visual inputs
Table 1: Within-subject metrics for all participants and all latent types on the test set Reported metrics are relative median rank & (MR) of retrieval on a set 500 samples, top-5 accuracy % (Acc) of retrieval on a set 500 samples. Chance level is at 50.0 and 1.0 for these two metrics respectively. These results were averaged across 50 retrieval sets, hence results are reported with a standard error of the mean (SEM) smaller than 0.01. The Dummy (D) model systematically predicts the mean latent representation of the training set and achieves chance level.
CLIP 257 x 768
VD-VAE
CLIP
CLS
AutoKL
MR
Acc
MR
Acc
MR
Acc
MR
Acc
Dummy
50.0
1.0
50.0
1.0
50.0
1.0
50.0
1.0
S1
9.4
13.8
29.9
3.0
15.1
8.4
24.9
3.9
$2
6.8
16.4
30.2
3.5
10.6
10.5
21.8
3.8
S3
7.8
13.6
28.5
3.1
11.0
9.9
26.0
3.3
We report video decoding results on the retrieval task in Table For all three participants of the Wen 2017 dataset, and for all four lypes of latent. representations considered, Ridge regression fitted within-subject achieves significantly above-chance performance. Besides, performance varies across participants, although well-performing participants reach good performance on all types of latents.
Results reported in Table 1 were obtained for a lag of brain volumes (i.e. 4 seconds since TR = seconds) and a window size of 2 brain volumes that were averaged together (see definitions in section 2.1). These parameters were chosen after running k-fold rross-validated grid search for lag values ranging from 1
hhttps:/nnieear: www.arannnggiiiiioi..o

---

Published in Transactions on Machine Learning Research (05/2025)
GT
Ret.
Ret. 2
Ret. 3
Ret. 4
Ret. 5
GT
Ret. 1
Ret. 2
Ret. 3
Ret. 4 Ret.
Image retrievals using predicted latent representations of CLIP 257 768 latents We use a model fitted on Subject 2 (S2) and predict the latent representation of unseen videos (test set). Ground truth (GT) images featured within the first 5 retrieved (Ret.) images are indicated with bold purple border. In given row, images which appear similar across columns are actually different frames of the same video clip. Images featuring human faces were blurred. More cases are available in supplementary
Figure 2:
Figure S4
5, window size ranging from to 3, and possible aggregation functions for brain volumes belonging to the same window (namely averaging and stacking). Figure S3 shows results using the averaging aggregation function for different values of lag and window size, averaged across participants. These results were obtained by stacking all runs of the training dataset, as opposed to averaging repetitions of the same video clip. The two approaches yielded very similar metrics. We give more details in section 4.3
Figure 2 shows retrieved images for Subject 2. Qualitatively, we observe that retrieved images often fit the theme of images shown to participants (with categorics like indoor sports, human faces, animals, etc.), yet with occasional failures.
4.2 Out-of-subject decoding and multi-subject training
As illustrated in Figure 3 models trained on one participant do not generalise well to other participants: using CLIP 257 x 768, the within-subject and out-of-subject median rank (MR) are respectively 8.0 and 17.2 average. However, functional alignment. allows reduce the median rank back to 11.1 on average. In particular, show that left-out participants do not need to have the same amount of available data training participants to benefit from their decoder: with only 30 minutes of data i.e. roughly 1000 samples left-out participants can reach performance which would have required roughly 100 minutes of data - i.e 3000 samples- in a within-subject setting.
In addition, we show that a single decoder trained on all functionally aligned participants can reach better results than a decoder trained on all un-aligned participants (MR is 7.7 against 8.6 averaged across subjects), and performs on par with each corresponding single-subject decoders.
Framework generality Note that, in Figure we chose the best performing participant (S2) as the ref- erence participant. We report all other combinations of reference and left-out participants in Supplementary Figures 56 and S7 and find that all effects persist for all combinations. In addition, supplementary Figures 58 and $9 show that these results hold for all types of latents. Furthermore, we replicate this experiment four participants from the Natural Scenes Dataset: decoding performance, reported in Tables S1 and S2 is higher than for the Wen 2017 dataset, probably due to using much more training data. Importantly, func- tional aligument allows the median rank to drop from 22.5 (baseline) to 11.5 on average in the out-of-subject setup for CLIP 257 x 768. In particular, it drops to 8.3 on average across left-out participants when S7 is used as reference. Finally, Figures [ST and [112 present all other sctups we ran for this study. In particular, they show that multi-subject aligned model (c.g. trained on S1 and S2) has better performance on aligned left-out. participants (e.g. S3) than single-subject model (e.g. trained on S2 only).
8

---

Published in Transactions on Machine Learning Research (05/2025)
A
Relative median rank
Train
Test
10
20
subject
6.6
19.3
Out-of-subject
16.7
Out-of-subject aligned
12.5
reduction
10.0
9.9
Multi-subject
S2 (ref)
sulte-s aligned
0.6
772
S3
Relative median rank
20.0
17.5
15.0
12.5
10.0
7.5
5.0
32
64
128
256
32
64
128
256
Total amount of training data in minutes
Figure 3: Effects of functional alignment on multi-subject and out-of-subject setups We report relative median rank 4 in all setups described in section 3.1 for CLIP 257 x 768. In all aligned cases, S1 and $3 were aligned onto S2. In all out-of-subject cases, test S1 and S3 onto decoder trained on $2. In all malli-subject cases, the decoder was trained on all data from all 3 participants. A. In this panel, all models (alignment and decoding) were trained on all available training data. Results for other latent types are available in Figure S8. B. In left-out S1 and S3, decoding performance much better when using functional alignment. to S2 (solid dark purple) than when using anatomical alignment only (solid pale purple). Porformance increases slightly as the amount of data used to align participants grows, but does not always reach levels which pale gray dots) when lot of training data is available. Training model multiple participants yields good performance in all 3 participants (dashed pale tcal) which can be further improved by using functional alignment (dashed dark leal). Results for other latent types are available in Figure S9
be achieved with single-subjecl model fitted in left-out participants (solid
Exploring computed inter-subject alignments To better understand how brain features are trans- formed by functional alignment, we show in Figure g how vertices from participant S1 are warped to fit those of participant S2. To this end, we colorize vertices in S1 using the MMP 1.0 atlas (Glasser et al. 2016) and use @81-$$2 transport each of the three RGB channels of this colorization to S2. Note that both participants' data lie on fsaverage5.
We that, in low data regimes, FUGW does not recover smooth inter-subject mapping of the cortical surface, but still manages to recover the cortical organization of the occipital lobe. A greater amount of data allows FUGW to reconstruct inter-subject mappings that are anatomically consistent in a much higher number of cortical areas such as the temporal and parietal lobes, and, unexpectedly, in the primary molor cortex as well. The prefrontal cortex and temporo-parietal junction (TPJ) seem challenging to map, perhaps due to greater inter-subject functional variability or lesser responsivity in those regions.
4.3 Influence of training set size and test set repetitions
Recent publications (Ozeelik & VanRullen 2023 Scotti et al. 2023 Tang et al. 2023 in brain decoding using fMRI have shown impressive results, but these results are obtained using unusually large datasots and signal-to-noise ratios (e.g. tens of hours of 7T fMRI per participant). To evaluate the importance of these two factors, report in Figure performance metrics for models trained with various amounts of data and

---

Published in Transactions on Machine Learning Research (05/2025)
MMP 1.0 atlas displayed on
S1
S1 + S2 10% data (28 min.)
S1 - S2 50% data (1 (144 min.)
S1 - S2 100% data (288 min.)
Lateral view
Posterior view
Figure 4: Visualizing functional alignments in the left hemisphere Vertices of the left-out participant (left column) are warped by FUGW. The result of this transport. is visualized on the reference participant (columns 2, 3, and 4). Fitting FUGW with increasing amounts of data gradually leads the inter-subject mapping to better respect the cortical organisation of multiple arcas, including non-visual ones. Note that all 3 models were fitted using the same number of iterations.
30
Relative median rank
2 runs mean
25
2 runs stack
20
15
10
16
32
64
128
Total duration of unique video segments (in minutes)
Number of averaged test runs
Figure 5: Influence of training set size and test set noise Relative median rank averaged across participants gets better more training data is used to fit the model (left). Interostingly, averaging brain volumes of 2 similar runs does not bring improvements compared to using just 1 run. Instead, stacking runs yields significant improvements. Note that training sets using 2 runs have twice as much data as those using run. Finally, these metrics are highly affected by the noise level the test set (right): averaging more runs in the test set yiclds better metrics despite using the same decoder.
fixed test set
tested with various amounts of noise. Firstly, using fixed test set, our results allow systematically estimate the quantity of training data needed to achieve given decoding performance. Interestingly, results show that stacking two runs displaying the same stimuli yields better results than averaging them. Besides, for given acquisition budget, showing different stimuli (as opposed to repcating stimuli) yiclds small but systematic performance improvements. Secondly, reported performance metrics only hold in lavorable signal-to-noise selups. Indeed, the test set associated with the Wen 2017 dataset comes with 10 runs for each video segment, which, when averaged together, greatly reduce the noise level. However, as reported Figure 5 when tested in real-life signal-to- noise conditions (i.e. only one run per test video clip), our models' performance degrades: when using CLIP latents, for each participant, it is approximately twice bad when averaging all 10 runs.
10

---

Published in Transactions on Machine Learning Research (05/2025)
5 Discussion
Impact The present work confirms the feasibility of using [MRI signals in response to natural images and videos to decode high level visual features (Nishimoto et al. 2011). It further demonstrates that it is possible to leverage these fMRI signals to estimate meaningful functional alignments between participants, and use them to transfer semantic decoders to novel participants. In particular, our study shows that decoding brain data from a left-out participant, i.c. a participant who was not used to train the decoder, can be substantially improved by aligning this left-out participant to a large reference datasct trained on large amounts of individual data to decode signals acquired in smaller neuroimaging studies, which typically record an hour or two of fMRI data for each participant (Madan 2022). We also find that training a brain decoder on multiple functionally-aaigned participants systematically improves decoding performance in these participants.
which a decoder was trained. Our method thus paves the way to using models
In addition, this study reports decoding accuracy in setups where participants are shown test stimuli for the first time, thus providing insight into how these models would perform in real-time decoding. While performance improves with the number of repetitions at test time, rcasonable decoding performance of semantics can be achieved with only one repetition in two out of three participants.
Note that decoders can also be used to decode brain activity for which it is hard to provide a good latent representation, for instance when the participant is sleeping, or when dealing with animals. Our method could be used to decode brain activity in left-out participants in such cases, using a decoder that had previously been trained on a large amount of labelled data. Lastly, by systematically quantifying decoding accuracy as a function of the amount of training data, the present work brings insightful recommendations as to what stimuli should be played in future [MRI datasets collecting large amounts of data in a limited number of participants. In the current setup (naturalistic movie clips acquired at 3T), training with diverse semantic content is more valuable than training with repeated content for fitting decoding models.
Limitations This work is first step towards training accurate semantic decoders which generalize across individuals, but subsequent work remains necessary to ensure the generality of our findings. Firstly, although the reported gains in out-of-subject setups are significant, the small mumber of participants present in the dataset under study requires replications on larger cohorts. However, to our knowledge, no other dataset has presented similar features to Wen et al. (0117). namely a large amount of data per partic- ipant and a large variely of video stimuli.
Secondly, our approach currently requires left-out participants to watch the same stimuli as reference par- ticipants. It is yet unclear whether functional alignment could bring improvements without this constraint. However, multi-subject decoding can probably help partially address this issue: since it is possible to train decoder on multiple participants and because not all of them have to watch the same movies, it is possible that a lot of different movies each seen by different participant used in the training set could be used as "anchors" for left-out individuals.
Finally, while restricting this study to linear models makes sense to establish baselines and ensure repli- cability, non-linear models have proved to perform as well (Scotti al., 2023), and constitute a natural improvement of this work.
Ethical implications Out-of-subject generalization an important test for decoding models, but it raises legitimate concerns. In this regard, this study highlights that signal-to-noise ratio still currently makes it challenging to very accurately decode semantics in real-time setup, and that non-trivial amount of data is needed per individual for these models to work. In particular, it would be interesting to see if recent work in perception decoding in MEG (Défosscz et al. 2022 Benchetrit et al. 2023) could be applicd to out-of-subject setups with a method similar to ours. Moreover, we stress that, while great progress has been made in decoding perceived stimuli, imagined stimuli are still very challenging to decode (Horikawa. & Kamitani (017). Nonetheless, it important for advances in this domain to be publicly documented. We thus advocate that open and reer-reviewed research the best way forward to safely explore the implications of inter-individual modeling, and more generally brain decoding.
11

---

Published in Transactions on Machine Learning Research (05/2025)
References
Alexandre Abraham, Fabian Pedregosa, Michael Eickenberg, Philippe Gervais, Andreas Mueller, Jean Kossaifi, Alexandre Gramfort, Bertrand Thirion, and Gacl Varoquaux. Machine learning for neu- roimaging with scikit-learn. Frontiers in Neuroinformatics, 8, 2014. ISSN 1662-5196. URL https:
wwww.frontien.
orggarticle/ll http://ceee/...39//mmmmm.a 3389/fninf
2014.00014
Emily J. Allen, Ghislain St- Yves, Yihan Wu, Jesse L. Breedlove, Jacob S. Prince, Logan T. Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, J. Benjamin Hutchinson, Thomas Nasclaris, and Kendrick Kay. A massive 7T [MRI dataset to bridge cognilive neuroscience and artificial intelligence. Nature Newroscience, pp. 1-11, December 2021. ISSN 1546-1726. doi: 10.1038 10.1038//41193-111 //www.nature
https::::
aundicra_abtesse a Cg_type Nature Research
Journals caimary_atypar Research Publisher: Nature Publishing Group Subject_terpp term: Cortex;Neural encoding:Object vision:Perception Subject term id: hateeeeuuneekeooooooggogeeeeerrr
Emily J. Allen, Ghislain St- Yves, Yihan Wu, Jesse L. Breedlove, Jacob S. Prince, Logan T. Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, J. Benjamin Hutchinson, Thomas Nasclaris, and Kendrick Kay. A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence. Nature Newroscience, (5(1):116 126, Jamuary 2022. ISSN 1526-1726. doi: 10.1038/ 10.1038/s41593- 021-00962-x. 12201000111 URL https://wwi nature. com/ /tt.://teeeeee111-0000000-333330 Group.
Number: Publisher: Nature Publishing
T. Bazeille, H. Richard, H. Janati, and B. Thirion. Local Optimal Transport for Functional Brain Template Estimation. In Albert C. S. Chung, James C. Gee, Paul A. Yushkevich, and Siqi Bao (eds.), Information Processing in Medical Imaging, Lecture Notes in Computer Science, pp. International Publishing. ISBN 778-3-030-20351-1 doi: 10.1007/978-3-030 9-000-00----00-001---1-1 18.
Cham, 2019. Springer
Thomas Bazcille, Elizabeth DuPre, Hugo Richard, Jean-Baptiste Poline, and Bertrand Thirion. An empir- ical evaluation of functional aligument using inter-subject decoding, Nearolmage, 245: 18683, December 2021. ISSN 1053-8119, doi: htw.:////...urommag.... 21861.118.2.2..11188 URL
com/
Yohann Benchetrit, Hubert Banville, and Jean-Rémi King, Brain decoding: toward real-time reconstruction of visual perception. arXiv preprint ar 1x1o:2310.19812, 2023.
Po-Hsuan (Cameron) Chen, Janice Chen, Yaara Yeshurun, Uri Hasson, James Haxby, and Peter J Ramadge. A Dedluced-Dimenssio [MRI Shared Response Model. In Advances in Neurul Information Processing Systems, volume 28. Curran Associates, Inc., 2015. URL httpt::////// paattps://pppprsss nips http:cccpppapppeeeee files/paper.
Zijiao Chen, Jiaxin Qing, Tiange Xiang, Wan Lin Yue, and Juan Helen Zhou. Seeing Beyond the Brain: Conditional Diffusion Model with Sparse Masked Modeling for Vision Decoding, March 2023a. URL
http://arxiv. http:://arxivorr//bbbba www./os//////ooooooooo..oo...e/t 2atps////21...e 06956 arXiv:2211 NNN:22211.06966 [cs).
Zijiao Chen, Jiaxin Qing, and Juan Helen Zhou Cinematic Mindscapes: High-quality Video Reconstruction from Brain Activity May 2023b. URL http:://aaxxi..rr//bbb////// 11675 arXiv:2305.11675 [cs).
Marco Cuturi. Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances, arXiv June 2013. doi: 10.48550/ 10.48550/arriv.1333 001550//aXXiv.106.00822
Alexandre Défossez, Charlotte Caucheteux, Jérémy Rapin, Ori Kabeli, and Jean-Rémi King. Decoding speech from non-invasive brain recordings, August 2022. URL http:::/axxiv.rr/bbbb..... 12266 W:Xiv:2208.12266 [cs, eess, q-bio].
Ma Feilong, Samuel A. Nastase, Guo Jiahui, Yaroslav O. Halchenko, M. Ida Gobbini, and James V. Haxby. The Individualized Neural Tuning Model: Precise and generalizable cartography of functional architoc- ture in individual brains, May 2022. URL hattps://ww www.wiiooiivvcommmo www.wbiwww..ooooteeoooeeeeeeeeee wworri..org/oottt///o100000/ 11w1://000...1/02222.. 15. 492022v1 Pages: 2022.05.15.492022 Section: New Results.
12

---

Published in Transactions on Machine Learning Research (05/2025)
Matteo Ferrante, Furkan Ozcelik, Tommaso Boccato, Rufin VanRullen, and Nicola Toschi. Brain Captioning: Decoding human brain activity into images and text, May 2023. URL
11560 arXiv:2305.1156 [cs].
Bruce Fischl. FreeSurfer.
Newrolmage, 62(2):774-781, August 2012. ISSN 1053-8118 8119. doi: 10.1016/j..euroimnngs 2012000..0....1.... URI. bttps://www.sciencee/ S105881112200899
Matthew F. Glasser, Stamatios N. Sotiropoulos, J. Anthony Wilson, Timothy S. Coalson, Bruce Fischl, Jesper L. Andersson, Junqian Xu, Saad Jbabdi, Matthew Webster, Jonathan R. Polimeni, David C. Van Essen, Mark Jenkinson, and WU-Minn HCP Consortium. The minimal preprocessing pipelines for the Human Connectome Project. Newrolmage, 80:105-124, October 2013. ISSN 1095-9572. doi: 10.1016/j www.onnag...2211......oo
Matthew F. Glasser, Timothy S. Coalson, Emma C. Robinson, Carl D. Hacker, John Harwell, Essa Yacoub, Kamil Ugurbil, Jesper Andersson, Christian F. Beckmann, Mark Jenkinson, Stephen M. Smith, and David C. Van Essen. A multi-modal parcellation of human cerebral cortex. Nature, 536(7615):111-17, August 2016. ISSN 1476-4687. doi: 10.1038/ 20.108/naatrrel8 hntuure18933. URL https:///wwwt nature nature18933 Number: 7615 Publisher: Nature Publishing Group.
G. H. Glover. Deconvolution of impulse response in event-related BOLD fMRI. Newrolmage, 944::116445 429, April 1999. ISSN 1053-8119.
doi: 10. 10.1006/nig. 1998.0419.
Zijin Gu, Keith Jamison, Amy Kuceyeski, and Mert Sabuncu. Decoding natural image stimuli from [MR data with a saeffcce-based convolutional network, March 2023. httpt:::/aarxi..og//bb////// 02409.
arXiv:2212.02409 [cs, q-bio].
Stephenie A Harrison and Frank Tong. Decoding reveals the contents of visual working memory in early visual areas Nature, 458(883)66632 -32,, 2009.
James V. Haxby, J. Swaroop Guntupalli, Andrew C. Comnolly, Yaroslav O. Halchenko, Bryan R. Conroy, M. Ida Gobbini, Michael Hanke, and Peter J. Ramadge. A common, high-dimensional model of the representational space in human ventral temporal cortex. Newron, 72(2):404 416, October 2011. ISSN 1097-4199. doi: 10.1016/j conurrnn2211...0 nwwron....000.20111.... 2011.08.026.
James V Haxby, J Swaroop Guntupalli, Samuel A Nastase, and Ma Feilong. Hyperalignment: Modeling shared information encoded in idiosyncratic cortical lopographies. eLife, 9ee56601, June 2020. ISSN 2050-084X. doi: 10.7554/eLife.5600. URL attps://doi.org// 755.00.55544/feeee 56601 Sciences Publications, Ltd.
Publisher: eLife
John-Dylan Haynes and Geraint Rees. Decoding mental states from brain activity humans. Nature rewiews neuroscience, -(3)5523,534, 2006.
Jun Kai Ho, Tomoyasu Horikawa, Kei Majima, Fan Cheng, and Yukiyasu Kamitani. Inter-individual deep image reconstruction via hierarchical neural code conversion. Newro/mage, 271:120007, May 2023. ISSN 20w.. 120007. URL
1053-8119. doi: arricle/jii/S105581199900
Tomoyasu Horikawa and Yukiyasu Kamitani. Generic decoding of seen and imagined objects using hierarchi- cal visual features. Nat. Commun., 8 8(15037):1-15, May 2017. ISSN 2041-1723. doi: 10.1038 221038/ncommss ccommmmmmmmmmmm 1037.
Ganit Kupershmidt, Roman Beliy, Guy Gaziv, and Michal Irani. A Penny for Your (visual) Thoughts: Self-Supervised Reconstruction of Natural Movies from Brain Activity June 2022. URL http://arxiv. org/abs/2206 2ttp//2222000554444 03544 arXiv:2206.033: 03544 [cs].
Benjamin Lahner, Kshitij Dwivedi, Polina lamshchinina, Monika Graumann, Alex Lascelles, Gemma Roig, Alessandro Thomas Gifford, Bowen Pan, SouYoung Jin, N. Apurva Ratan Murty, Kendrick Kay, Aude Oliva, and Radoslaw Cichy. BOLD Moments: modcling short visual events through a video fMRI datasct and metadata, March 2023. URL httpp:://ww/ http///wwbbioxxi...../. wttpr//w.orr/coottt//0.....000.0 1101/////000.11/222331 03. 12. 530887v1 Pages: 0123.03.12.5300 Section: New Results.
13

---

Published in Transactions on Machine Learning Research (05/2025)
Amanda LeBel, Lauren Wagner, Shailee Jain, Ancesh Adhikari-Desale Bhavin Gupta, Allyson Morgenthal, Jerry Tang, Lixiang Xu, and Alexander G. Huth. A natural language IMRT dataset for voxelwise encoding models. Scientific Data, 10(1):555, August 2023. ISSN 2052-4463. doi: 10.1038/s415977 /4977/132--2-222277 URL
nature. com http://cww...oooaarrr..////////2
Number: Publisher: Nature Publishing
Group.
Christopher R. Madan. Scan Once, Analyse Many: Using Large Open-Access Neuroimaging Datasets to Understand the Brain. Neuroinformatics, 20(1):109-14) 137, January 2022. ISSN 1559-0089, doi: 10.1007/ s12021-021-0951996.
Weijian Mai and Zhijun Zhang. UniBrain: Unify Image Reconstruction and Captioning All in One Dif- fusion Model from Human Brain Activity, August 2023. URL
07428
arXiv:2308.07428 [cs].
Tom M Mitchell, Rebecca Hutchinson, Radu S Niculescu, Francisco Pereira, Xuerui Wang, Marcel Just, and Sharlene Newman. Learning to decode cognitive states from brain images. Machine learning, 57:145-175, 2004.
Shinji Nishimoto, An T. Vu, Thomas Naselaris, Yuval Benjamini, Bin Yu, and Jack L.. Gallant. Reconstruct- ing visual experiences from brain activity evoked by natural movics. Current Biology, 21119):661-1164 October 2011. ISSN 0960-9822. doi: 2..1066/..cbb.201108..00 URL https://vww.wr ncbi nnm. nih. nih.gov/ ppt//artiicles/PMc322551
Furkan Ozcelik and Rufin VanRullen, Natural scene reconstruction from fMRI signals using generative latent diffusion, June 2023. URL
05334 arXiv:2303.0533 [cs, q-bio].
Fabian Pedregosa, Gaël Varoquaux, Alexandre Gramfort, Vincent Michel, Bertrand Thirion, Olivier Grisel, Mathicu Blondel, Peter Prettenhofer, Ron Weiss, Vincent Dubourg, Jake Vanderplas, Alexandre Passos, David Cournapeau, Matthieu Brucher, Matthieu Perrot, and Edouard Duchesnay. Scikit-learn: Machine Learning in Python. Journal of Machine Learning Research, 12(85):2825 2830, 2011. ISSN 1533-7928.
URL http://jmlr. www.erorr//ppaeee.eeee httppppss/////dprepeeeppeee hammmll......mmm.....mm.....m
Erin M. Phillips, Kirsten D. Gillette, Daniel D. Dilks, and Gregory S. Berns. Through a Dog's Eyes: fMRI Decoding of Naturalistic Videos from the Dog Cortex. JoVE (Journal of Visualized Experiments), (187):
e64442, September 2022. ISSN 1940-087X. doi: 10.3791 /64442. URL https: //www. jove. com/fr/v/
64442/throoghhdo--ssa -ar-ee----ii--deeeddnnnnaaaaaare
Ana Luisa Pinho, Alexis Amadon, Torsten Ruest, Murielle Fabre, Elvis Dohmatob, Isabelle Denghien, Chan- tal Ginisty, Séverine Preuwe-Desmidl. Séverine Roger, Laurence Laurier, Véronique Joly-Tcstault, Gaölle Médiouni-Cloarec, Christine Doublé, Bernadette Martins, Philippe Pinel, Evelyn Eger, Gaël Varoquaux, Christophe Pallier, Stanislas Dehaene, Lucie Hertz-Pannier, and Bertrand Thirion. Individual Brain Charting, high-resolution fMRI dataset for cognitive mapping. Scientific Data, 5(1):180105, June 2018. ISSN 2052-4463. doi: 4038//daat..001 105. Number: Publisher: Nature Publishing Group.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning Transferable Visual Models From Natural Language Supervision, February 2021. URL http://arxiv. arXiv:2103.00020 [cs].
org/abs//103300002
Hugo Richard, Luigi Gresele, Aapo Hyvärinen, Bertrand Thirion, Alexandre Gramfort, and Pierre Ablin. Modeling Shared Responses in Neuroimaging Studies through MultiView ICA, December 2020. URL
httpp::/aaxxv..rrg//// ht///bbs/2000 06635 ar arXiv:2006.06633 [es, stat].
Emma C. Robinson, Saad Jbabdi, Matthew F. Glasser, Jesper Andersson, Gregory C. Burgess, Michael Harms, Stephen M. Smith, David C. Van Essen, and Mark Jenkinson. MSM: new flexible framework for Multimodal Surface Matching. Neurolmage, 100:414-426, October 2014. ISSN 1095-9572. doi: 10.1016/j. neuroimage.20 4.05.069.
14

---

Published in Transactions on Machine Learning Research (05/2025)
Paul S. Scotti, Atmadeep Banerjee, Jimmic Goode, Stepan Shabalin, Alex Nguyen, Ethan Cohen, Aidan J. Dempster, Nathalie Verlinde, Flad Yundler, David Weisberg, Kenneth A. Norman, and Tanishq Mathew Abraham. Reconstructing the Mind's Eye: IMRl-ttoImaage with Contrastive Learning and Diffusion Priors, May 2023. URL http://arxiv. org/abs/ 2alaab//305. 2305 18274 MrXiv:2305.18217 [cs, q-bio].
Thibault Séjourné, Trangois-Xavier Vialard, and Gabriel Peyré. The Unbalanced Gromov Wasserstein Dislance: Conic Formulation and Relaxation. arXiv:2009.04266 fmalh, stat], June 2021. URL http: arXiv: 2009.04266.
//arxiv. //arxiv.org/aart wwgg/abs/209.... 04266
Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity, March 2023. URL http://wwwwwwoooooooooo/c///o///
517004v3
Pages: 2022.11.18.511
Section: New Results.
Jerry Tang, Amanda LcBel, Shailee Jain, and Alexander G. Huth. Semantic roconstruction of continuous lan-
guage from non-invasive brain recordings. Nalure Newroscience, 26(5):858 866, May 2023. ISSN 1546-1726.
doi: 10.1008//a... 10.1038/s41593 (s1593-02223 023-01304-9. URL
Number: Publisher: Nature Publishing Group.
Armin Thomas, Christopher Ré, and Russell Poldrack. Self-Supervised Learning of Brain Dynam- ics from Broad Neuroimaging Data. Advances in Neural Information Processing Systems, 35:21255 21269, December 2022. URL https: http://owoceddn....oooorrr...ooe wtppaaae....rppppppppapeeeeeeeee http://appp..ll////pp//////////a 8600a9df 00000000a00000000.00000000000000
html
Alexis Thual, Quang Huy Tran, Tatiana Zemskova, Nicolas Courty, Rémi Flamary, Stanis- las Dehaene, and Bertrand Thirion. Aligning individual brains with fused unbalanced Gro- mov Wasserstein. Advonces in Neural Information Processing Systems, 35:21792-21804,
http:://ppooeeddnnn....iiireeeee cww.aps.c.ppppp.ppppr........ December 2022. URL wwwtract- http:r/pp.cppppppp...e////p/pp/p fwtpe//paapprr/02///aa.
html
Chong Wang, Hongmei Yan, Wei Huang, Jiyi Li, Yuting Wang, Yun-Shuang Fan, Wei Sheng, Tao Liu, Rong Li, and Huafu Chen. Reconstructing rapid natural vision with IMRI-conditional video generative adversarial network. Cerebral Corter, 32(20):4502-4511 October 2022. ISSN 1047-3211. doi: 10.1093/
cercor/bhab498. URT. htttss://oi..oooo wrg. 1ttp://...99///eeeeeee
Haiguang Wen, Junxing Shi, Yizhen Zhang, Kun-Han Lu, Jiayue Cao, and Zhongming Liu. Data for Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision Tests, September 2017. URL
httpss//purr... purdue. wttp:puuuuuuuuuuuuupeepe edu.ppublications//88000
Haiguang Wen, Junxing Shi, Yizhen Zhang, Kun- Han Lu, Jiayue Cao, and Zhongming Liu. Neural Encoding and Decoding with Deep Learning for Dynamic Natural Vision. Cerebral Corter (New York, N.: 1991), (8(12):4136-4100 December 2018. ISSN 1460-2199 999. doi: 10.1093/cercor////
15

---

Published in Transactions on Machine Learning Research (05/2025)
A Appendix
A.1 Data pre-processing
We strive to minimally preprocess acquired BOLD signal. To this end, we detrend acquired BOLD signal (i.e. we remove cosine drifts) and linally standardize voxels' Limecourses for each run, as shown in Figure ST
Morcover, when decoding the latent representation of given image, we use brain volumes which have been acquired after the image's onset. Figure S2 illustrates this idea, and introds the concepts of window size (i.e. the number of brain volumes
use) and lag (i.e. the time difference between the onset of the image to be decoded and the first brain volume used to decode it). Values for both of these hyper-paramcters were obtained through 5-fold cross- validate grid search over samples of the training set. We report these results in Figure S3
Run design matrix
Raw data
Detrended and standardized data
50000
4
50
Brain volumes
40000
2
100
35000
0
150
200
25000
20000
constant
100
200
300
400
500
0
100
200
300
400
500
drift onit ohif drift dnitt ohift oni
dr
Brain volumes
Brain volumes
Figure S1: Pre-processing of the Wen 2017 dataset For each participant and each run, in each vertex, regress out parts of the signal which can be linearly explained by the design matrix represented the left, which models cosine drifts of the BOLD signal. The two graphs to the right show time-courses in 5 vertices across 2 different runs before (left) and after (right) they have been pre-processed.
lag
window size
Brain volumes
time
Movie
frames
prediction
Figure S2: Lag and window size In order to decode a movie frame which was seen at time f, one can use brain volumes which were acquired further in time. This delay is referred to as the lag. Moreover, one can use several brain volumes decode a given movie frame. The number of brain volumes used is called the window size, Images featuring human faces were blurred.
16

---

Published in Transactions on Machine Learning Research (05/2025)
Clip Vision 257
VD-VAE
Clip Vision CLS
SD AutoKL
Relative median rank
Window size (in TRs 2 seconds)
17.8
8.6
9.6
19.7
34.3
43.1
39.8
40.9
44.6
48.3
23.0
13.7
14.9
23.9
35.3
33.3
25.1
25.2
32.3
42.1
-40
20
39.7
22.9
9.4
9.0
20.1
48.1
44.5
41.2
40.9
45.5
42.2
28.7
15.0
13.5
24.5
44.7
34.3
24.3
25.0
33.1
10
2
3
3
0
1
4
Lag (in TRs of 2 seconds)
Figure S3: Relative median rank of predicted latents averaged across participants for various time lags and window sizes
17

---

Published in Transactions on Machine Learning Research (05/2025)
A.2 Retrieving images using predicted latent representations
Predicted latent representations can be compared to that of images in a retrieval set. In Figure S4 for each image shown to the participant during the test phase, we print the five images from the retrieval set whose latent representation is the closest to prodicted latents. We see that semantics are often preserved.
GT
Ret.
Ret.
Ret.
Ret. 5
GT
Ret.
Ret. 4 Ret.
Figure $4: Image retrievals using predicted latent representations of CLIP 257 x 768 latents We use a model fitted on Subject (S2) from the Wen 2017 dataset and predict the latent representation of unseen videos (test set). Ground truth (GT) images featured within the first retricved (Rct.) images are indicated with a bold purple border. In given row, images which appear similar across colurnss are actually different frames of the same video clip. Images featuring human faces were blurred.
18

---

Published in Transactions on Machine Learning Research (05/2025)
A.3 Results for every combination of reference participant and left-out participant
copy of Figure 3 [rom the main pages of this paper. It illustrates the main effects reported in our study, namely that (1) functional alignment yields better performance than anatomical alignment when transforring a semantic decoder to left-out individuals, (2) it is possible to train such decoders on multiple participants and (3) this last setup works best when participants are aligned.
Figure S5
Figure S5 only shows these results when participant of the Wen 2017 dataset is used as the reference participant. Therefore, add Figures [S6 and [S7] which illustrate that all results hold regardless of what participant of the cohort is used reference or left-out participant.
A
Relative median rank
Train
Test
6
10
14
16
20
Within-subject
6.8
Out-of-subject
-40% reduction
Out-of-uubect aligned
Multi-subject
52 (ref)
Multi-subject aligned
b.6
ZZZ
S3
B
S1
S3
Relative median rank
20.0
17.5
15.0
12.5
10.0
5.0
32
64
128
256
32
64
128
256
Total amount of training data in minutes
Figure $5: Effects of functional alignment on multi-subject and out-of-subject setups using participant 2 the reference participant We report relative median rank J in all setups described in section 3.1 for CLIP 257 x 768. In all aligned cases, S1 and S3 aligned onto S2. In all out-of-subject cases, we test S1 and S3 onto a decoder trained on S2. In all malli-subject cases, the decoder was trained on all data from all participants. A. In this panel, all models (alignment and decoding) were trained on all available training data. Results for other latent types are available in Figure 58] B. In left-out S1 and S3, decoding performance is much better when using functional alignment to $2 (solid dark purple) than when using anatomical alignment. only (solid pale purple). Performance increases slightly as the amount of data used to align participants grows, but does not always reach levels that can be achieved with a single- participant model fitted in left-out participants (solid pale gray dots) when lot of training data is available. Training model on multiple participants yields good performance in all 3 participants (dashed pale teal) which can be further improved by using functional alignment (dashed dark teal). Results for other latent types are available in Figure 59
19

---

Published in Transactions on Machine Learning Research (05/2025)
A
Relative median rank
Train
Test
6
10
12
14
16
18
20
9.4
Within-subject
7.8
Out-of-subject
17.0
Out-of-ubject aligned
10.3
9.9
Multi-subject
NNN
S1 (ref)
9.4
S2
subjett aligned
7.5
B
52
Relative median rank
20.0
17.5
15.0
12.5
10.0
5.0
32
64
128
256
32
64
128
256
Total amount of training data in minutes
Figure S6:
Effects of functional alignment on multi-subject and out-of-subject setups using participant the reference participant
A
Relative median rank
Train
Test
6
10
12
14
16
18
20
9.4
Within-subject
7.8
18.0
Out-of-subject
13.6
Out-of-subject aligned
9.9
Multi-subject
SNN
S1
9.7
S2
Multi-subject aligned
7.2
S3 (ref)
B
51
Relative median rank
20.0
17.5
15.0
12.5
10.0
7.5
5.0
32
64
128
256
32
64
128
256
Total amount of training data in minutes
Figure S7:
Effects of functional alignment on multi-subject and out-of-subject setups using participant 3 the reference participant
20

---

Published in Transactions on Machine Learning Research (05/2025)
A.4 Results for every type of latent representation
In this section, exlend the claims made in Figures A, for other latent representations, namely VD-VAE, CLIP CLS and AutoKL. Figures S8 S9 and S1O extend Figures 3 A, B and respectively, showing that observed effects are present regardless of the chosen latent representation. All of these figures were obtained using the Wen 2017 dataset.
B and 5 by showing that. these results hold
Clip 257 768
VD-VAE 28.5
Clip CLS
AutoKL
Within-subject
26.0
6.8
10.6
Out-of-subject
13.3
39.1
26.
30.9
37.4
Out-of-subject aligned
10.6
1622
30.5
9.9
24.3
15.0
25.2
Multi-subject
20.5
21.0
ANN S1
32.3
Multi-subject aligned
26.4
52 (ref.)
10
15
20
20
30
40
10
20
30
20
30
40
Relative median rank
Figure S8: Effects of alignment For any type of latent representation, out-of-subject decoding performance, measured through relative median rank 4. greatly improves when participants are functionally aligned. Training decoders on multiple participants also works better when participants are aligned. These results were averaged across 50 retricval sets all these metries are reported with a standard error of the mean (SEM) smaller than 0.01.
Clip 257 x 768
VD-VAE
Clip CLS
AutoKL
Chance
Within-subjee
20.0
Out aligned
Out-of-subject
40
S1
17.5
Multi-subject
15.0
Multi aligned
20
12.5
Relative median rank 4
10.0
10
7.5
32
64
256
64
256
256
256
22.5
20.0
40
17.5
S3
30
12.5
10.0
10
7.5
32
64
128
64
256
64
256
64
256
Total amount of training data in minutes
Figure S9: Performance increases slightly with more alignment data For any type of latent repre- sentation, out-of-subject decoding performance greatly increases with functional alignment even in low data regimes. In high data regimes, out-of-subject decoding does not work as well as fitting single-subject multi-subject models.
21

---

Published in Transactions on Machine Learning Research (05/2025)
Clip 257 768
VD-VAE
Clip CLS
AutoKL
30
Chance
run
25
52
51
runs mean
40
20
runs stack
30
53
15
Relative median rank
128
16
16
Total duration of unique video segments seen (in minutes)
30
50
25
40
20
30
15
52
10
10
4
8
2
8
2
8
Number averaged test runs
Figure S10: Scaling studics for all latents For any type latent reprosentation, decoding performance increases linearly with exponentially more data. It. also seems that, when acquiring data at 3T or more, not repeating stimuli yields the best results. At test time, although repeating stimuli allows get better metrics, retrieval performance with only one repetition already reasonable in 2 out of participants of the Wen 2017 dataset.
22

---

Published in Transactions on Machine Learning Research (05/2025)
A.5 Decoding results for all setups
On top of experiments reported in the main pages of this paper, we have tested lot of different training and tests sets. In this section, we report the Relative Median Rank & for all 97 training sets and all 63 test sets, and CLIP latent representations. Training sets include all possible singlo-subject, multi-subjoct unaligned and multi-subject aligned cases. Test sets include all possible with-subject, left-out unaligned and left-out aligned cases. Every time, all available training sessions are used for training the decoder. However, we vary the amount of data used to train the alignments, for both training and test sets.
We report detailed results for CLIP 257 768 and CLIP CLS in Figures S11 and S12
respectively.
In particular, these figures report combinations our setups of interest which were not mentioned in the main text. We find of particular interest the multi-subject out-of-subject setup, in which the decoder has been trained on two (un)aligned participants and tested on a third (un)aligned one.
23

---

Published in Transactions on Machine Learning Research (05/2025)
Figure S11: Relative median rank | for CLIP 257 x 768 latents in single- and multi-subject training sets, with and without alignment, tested on within- and across-participe particpaatis setups with and without alignment. These results were averaged across 50 retrieval sets all these metrics are reported with a standard error of the mean (SEM) smaller than 0.01.
24

---

Published in Transactions on Machine Learning Research (05/2025)
Figure S12: Relative median rank J for CLIP CLS latents in single- and multi-subject training sets, with and without alignment, tested on within- and across-participants setups with and without alignment. These results were averaged across 50 retrieval sets all these metrics are reported with a standard error of the mean (SEM) smaller than 0.01.
25

---

Published in Transactions on Machine Learning Research (05/2025)
A.6 Replication the Natural Scenes Dataset
We replicale our main experiment using data from the Natural Scenes Dataset (Alln el al. 2022) This dataset comprises 8 participants who each see 10 000 images 3 times, thus leading to total of 30 000 trials per participant. For cach participant, this data acquired in 40 sessions of 60 minutes each. For cach participant, there a total of 1 000 images which are shared with other individuals i.. other individuals will see them too and 9 000 which are exclusive i.c. other individuals will not see them. We sub-selected all participants who had completed all 30 000 trials, namely participants 1, 2, 5, and For each selected participant, split their 30 000 trials in two sets: all exclusive images are grouped in the decoding set and all shared images are grouped in the alignment set. We further split the decoding set into disjoint sets of images for training and testing individual decoders, whose performance is reported in Table ST Alignments sets are used to compute functional aligmments between individuals. Besides, we used pre-computed beta coefficients computed with GLM denoise on fsaverage? (Fischl 2012) and openly available online. We down- sampled this data to fsaveraged which simply amounts keeping only the first 10 242 array elements each hemisphere.
Eventually, show that decoders tested left-out individuals work consistently and significantly better when left-out participants are functionally aligned rather than simply anatomically aligned to the reference participant, as reported in Table 52
Table S1: Within-subject metrics for all NSD participants and all latent types on the test set
Reported metrics are relative median rank + (MR) of retrieval on a set 500 samples, top-5 accuracy t (Acc) of retrieval on a set of 500 samples. Chance level is at 50.0 and 1.0 for these metrics respectively, These results were averagod across 50 retrioval sots, hence results are reported with standard error of the
(SEM) smaller than 0.01.
CLIP 257
768
VD-VAE
CLIP
CLS
AutoKL
MR
Acc
MR.
Acc
MR
Acc
MR
Acc
S1
3.6
26.6
23.0
4.6
4.6
19.1
30.5
1.8
$2
6.0
17.6
22.1
4.4
6.9
13.8
33.6
1.5
S5
4.6
19.9
26.0
3.7
4.3
19.7
31.5
2.5
S7
4.0
24.4
23.4
5.4
5.5
18.5
24.5
4.3
26

---

Published in Transactions on Machine Learning Research (05/2025)
Table S2: Across-subject metrics for all NSD participants and all latent types on the test set We report the decoding performance of decoders trained on a reference participant and tested on a left-out participant who was anatomically aligned (A) or functionally aligned (A+F). The reported metric is the relative median rank (MR) of retrieval retrieval sets, hence results reported with standard error of the mean (SEM) smaller than 0.01. One sees that functionally aligned data is always better decoded than anatomically aligned data. In particular, when S7 as the reference subject, functional alignment helps divide the median rank by 3 for CLIP latents.
set of 500 samples. These results were averaged across 50
CLIP 257
768
VD-VAE
CLIP
CLS
AutoKL
Reference
Left-out
A
A+F
A
A+F
A
A+F
A
A+F
$2
20.9
10.7
35.2
24.5
28.3
13.4
42.5
37.1
S1
S5
32.2
13.6
43.1
32.0
33.4
12.5
43.9
37.7
S7
33.5
14.7
40.8
30.7
36.1
17.1
45.0
37.0
S1
18.3
10.4
37.0
30.4
24.4
14.4
35.9
37.1
$2
$5
29.3
12.0
40.3
34.1
30.8
11.2
42.4
39.6
S7
27.9
14.6
40.2
32.8
32.1
17.9
41.8
38.9
S1
29.2
13.0
43.4
34.4
33.8
14.4
37.8
33.8
$5
S2
26.2
9.8
40.1
30.1
31.4
11.2
36.0
36.2
S7
29.8
14.1
41.7
32.0
34.4
17.3
40.0
34.9
S1
27.7
7.7
43.1
30.1
32.3
11.5
40.6
26.5
S7
$2
23.4
8.6
38.9
26.4
28.6
13.2
37.8
29.1
S5
27.5
8.8
41.6
31.2
30.4
9.8
43.4
30.9
27


Reviews:
[{"review_id": "IsSWBMvVhn", "reviewer": "Reviewer_bDCE", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "1. Strengths:\n\n1.1 The paper basically describes how the proposed method works.\n\n1.2 The paper claims that it can make models trained on large amounts of data work on certain \"outliers points\". This paper focuses on good problems.\n\n2. Weaknesses:\n\n2.1 The predicted latent representation does not seem to be very accurate. In fact, from Figure 2, I see that by retrieving the images with the latent representation that is closest to the predicted latent representation. The image seen by the participant can't be recovered(predicted). Especially the Eiffel Tower prediction. Also, from the Figure S4 of appendix, the predicted latent representation of dog seem to be to recovered to a human.\n\n2.2 The overall framework has practical problems, assuming that in the future work this type of overall technical route can obtain a very accurate predicted latent representation, in other words, even though the problem in 2.1 is perfectly solved by other work in the future(but the problem don't be solved in this paper), the framework can only guess what a person has seen through retrieval. Once what a person has seen is not in the database, then logically speaking, it is impossible to retrieve what the person has seen. If it is impossible to guess what humans actually see due to the content of the database, what is the meaning of predicting latent representation? If it is meaningful, please provide specific experimental scenarios and experimental data to illustrate the usage of the predicted latent representation.  I can give a solution as a reference (if it is wrong, please point out my problem), for example, using the generative model to amplify the latent variables decoded from brain features, and then map the amplified new latent variables to the image space, which can avoid the problem of missing content in the database.\n\n2.3 Out-of-subject is hard to defined, How do the work define what kind of participants are reference participants and what kind of participants are left-out participants? Is it based on artificial engineering experience? Or is there a physiological basis? Please explain in the paper.\n\n2.4 In fact, even if the problem in 2.3 is perfectly defined, the alignment model will not work when the number of left-out participants is small enough. So there is a problem here. This paper should analyze the impact of the ratio of Left-out and referecen-out on the alignment model, and quantify this impact into the final evaluation index.\n\n2.5 The encoding and decoding model part and the alignment model part of this paper are trained separately, so there is error accumulation, which will make the work not robust. The 3 human participants in the first dataset and 8 human participants in the second dataset should be expanded to prove that the method's robustness.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "1. The proposed functional alignment of the brain feafures across participants boosts visual semantics decoding performance in left-out participants, which have a limited amount of data compare to reference participants.\n\n2. Evaluated on a retrieval task, compared to the anatomically-aligned baseline, the paper halves the median rank in out-of-subject setups.\n\n3. The paper show this method aligns neural representations in accordance with brain anatomy.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "1. I suggest replacing the brain decoders in the figure annotation in figure 1 with model A and model B to improve readability.\n2. The functional data in 2.1 Functional alignment  need to be more detailed description to improve readability.\n3. Out-of-subject is hard to defined, How do the work define what kind of participants are reference participants and what kind of participants are left-out participants? Is it based on artificial engineering experience? Or is there a physiological basis? Please explain in the paper.\n4. This paper should analyze the impact of the ratio of Left-out and referecen-out on the alignment model, and quantify this impact into the final evaluation index.", "broader_impact_concerns": "Once the problem is perfectly solved in the future, can we read what people see from their fMRI? Will this cause privacy issues? Furthermore, visual signals can also be changed into auditory signals, tactile signals, and taste signals. So can we recover the external signals that people are exposed to through fMRI or other physiological data?", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ohy72tLWoh", "reviewer": "Reviewer_qt2z", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is easy to read and the authors provide a technique to boost out-of-subject performance in the limited data regime. In general, the claims seem validated. However I defer to the AC/other reviewers on the generality of the datasets used in the paper as I am not an expert on fMRI data.\n\nThere are a few improvements that can be made in presentation of the paper or claims. See below for a list of requested changes, the authors can consider updating the draft in response to the requested changes.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper deals with the task of mapping brain signals to neural network representations. In particular, the brain signals are a dataset of $X \\in R^{N \\times V}$ and the target neural network representations $Y \\in R^{N \\times M}$, where $N$, $V$ and $M$ are the number of brain scans, number of voxels and representation size. A linear mapping from brain signals to NN representations is learned using ridge regression.\n\nThe paper mainly focuses on the \"out-of-subject\" setup where the decoder is learned on one subject and then applied/transferred to 1 or more “out-of-distribution” subjects. They rely on (Thual et al 2022) to learn an alignment transformation between brain signals of the out-of-subject $X \\in R^{M \\times V}$ and in-subject $X \\in R^{N \\times V}$. One can then apply the same “in-subject” decoder to predict NN representations.\n\nResults are shown on Wen et al 2017 in the main paper and Allen et al 2021 (Natural Scenes Dataset) in the supplementary. The authors initially validate the linear decoder after reporting in-distribution metrics for three subjects S1, S2 and S3. They choose S2 as the in-distribution subject and study the transfer on S1 and S3. The main contributions are:\n\n* The alignment approach is better than learning a separate decoder directly on out-of-subject in the limited data regime.\n* Their approach is better than applying the \"within-subject\" decoder on \"out-of-subject.\"\n* Further, they show that training a single decoder on aligned data, works better than training a single decoder on unaligned data.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "* Some changes can be made to the structure of the paper. The paper combines the background work (Section 2.1 and Section 2.2) with their contributions (Section 2.3). It is better to separate this, so the readers can understand the exact contribution quickly.\n* The experimental setup (Section 2.4 - Section 2.6) is also merged with the methods section. The authors can move the experimental setup to a new section.\n* The functional alignment seems extremely useful, in cases where the “left-out” subjects do not have target latent representations. $Y \\in R^{n \\times m}$. The authors can consider adding some discussion about this to the main text.\n* There may be some potential overclaims. “Compared to the baseline, functional alignment across participants boosts visual semantics decoding performance in left-out participants, especially when the latter have a limited amount of data”. From the paper’s results, it does not boosts decoding performance in left-out participants when there is enough data. For example, In Fig 3. A, S1 and S3 achieve within-subject performance of 9.4 and 7.8 as compared to 12.5 and 10.0 with functional alignment. So if one has enough data, one can directly train decoders on 'out-of-subject\" data. Please fix these in the introduction and other places.\n* What is exactly v in the functional alignment setting? Is it still 10242?\n* The paper converts cosine similarity to a retrieval-based rank. Why not report mean cosine similarity between the predicted and target clip representations directly?\n* What metric is used to tune the hyperparameters in the cross-validated grid search?\n* In various sections of the paper, the authors use the minutes notation, which may be standard in neuroscience. It is also helpful to convert this to a number of examples. So that, general readers from the machine learning community can get a sense of the scale of the data.\n\nNot Critical:\n* Fig 5 is crowded. Just displaying S1 may be sufficient.\n* Replace Model A and Model B with low-level and high-level decoder.\n* Both CLIP and VD-VAE can retain both high-level and low-level representations of the input. So I'm not 100% sure of the low-level/high-level demarcation.", "broader_impact_concerns": "No", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Le88IdnlMm", "reviewer": "Reviewer_Ba96", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The goals of the paper are focused, and the approach is clear and intuitive. In some ways, this paper is not unlike machine translation, where each subject speaks its own language. The paper indeed demonstrates feasibility of using multiple subjects to improve an fMRI decoder.\n\nThe authors are pretty clear about the limitations and some weaknesses. The data requirements are heavy. Each subject has to watch the same videos and a separate alignment model.\n\nEvaluation could be better. Retrieval requires a database of existing latents, and it would be good to have some type of downstream task that only uses the latents.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper introduces a new system to decode fMRI scans of subjects into latent variables. The primary benefit of this model is that it generalizes across subjects via an alignment model. This is a way of getting more data and introducing noise which makes the model more robust and produce stronger latents. These latents were evaluated using a retrieval task.", "claims_and_evidence_yn": "Yes", "claims_explanation": "", "audience_yn": "Yes", "audience_explanation": "", "requested_changes": "More ablations could be done around the loss terms or different encoders.\n\nI would be curious to see another evaluation task. Perhaps zero shot text captioning from the CLIP embeddings? Or decoding images from the VAE?", "broader_impact_concerns": "It could be a serious privacy violation to decode ones thoughts via fMRI.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The strengths of this paper include
1) The proposed functional alignment method is novel and well-motivated.
2) The quantitative results are solid, showing superior decoding accuracy compared to baseline models.
3) The paper is well-written and clear, making complex ideas accessible.

This paper has been reviewed by two experts: one recommends acceptance, while the other leans toward rejection. The Associate Editor (AE), who is also an expert in this field, has evaluated the concerns raised.
Although one reviewer remains concerned that the latent representations may not be learned effectively enough, the task addressed in this paper is inherently complex and poses challenges that likely require broader community efforts beyond the scope of this work.

Overall, the paper supports its claims with empirical results and holds potential interest for the neuroscience community. It is well-structured, self-contained, and in good shape for acceptance. The authors should be encouraged to improve the camera-ready version by incorporating the reviewers' comments.
Decision: Accept (poster)