Toward Generalizing Visual Brain Decoding to Unseen Subjects
Authors: Xiangtao Kong
Kexin Huang
Ping Li
Lei Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: Visual brain decoding
fMRI - image retrieval
Generalizing to unseen subjects
Primary area: applications to neuroscience & cognitive science
TL;DR: The study explores the generalization capability of visual brain decoding across unseen subjects, showing that the generalization can be enhanced by increasing the training subjects and affected by the subject similarity.

Abstract: Visual brain decoding aims to decode visual information from human brain activities. Despite the great progress, one critical limitation of current brain decoding research lies in the lack of generalization capability to unseen subjects. Prior work typically focuses on decoding brain activity of individuals based on the observation that different subjects exhibit different brain activities, while it remains unclear whether brain decoding can be generalized to unseen subjects. This study aims to answer this question. We first consolidate an image-fMRI dataset consisting of stimulus-image and fMRI-response pairs, involving 177 subjects in the movie-viewing task of the Human Connectome Project (HCP). This dataset allows us to investigate the brain decoding performance with the increase of participants. We then present a learning paradigm that applies uniform processing across all subjects, instead of employing different network heads or tokenizers for individuals as in previous methods, so that we can accommodate a large number of subjects to explore the generalization capability across different subjects. A series of experiments are conducted and we have the following findings. First, the network exhibits clear generalization capabilities with the increase of training subjects. Second, the generalization capability is common to popular network architectures (MLP, CNN and Transformer). Third, the generalization performance is affected by the similarity between subjects. Our findings reveal the inherent similarities in brain activities across individuals. With the emergence of larger and more comprehensive datasets, it is possible to train a brain decoding foundation model in the future. Codes and models can be found at https://github.com/Xiangtaokong/TGBD}{https://github.com/Xiangtaokong/TGBD.

Published as a conference paper at ICLR 2025
TOWARD GENERALIZING VISUAL BRAIN DECODING TO UNSEEN SUBJECTS
Xiangtao Kong ; Kexin Huang Ping Li ! Lei Zhang If 'The Hong Kong Polytechnic University (xiangtao. kong, kexin0 huang) polyu.hk ping2. pepolpppllle..e edw..ddd.... edu.hk, cs cslzhang@comp. pol yu. edu .du.hk
ABSTRACT
Visual brain decoding aims to decode visual information from human brain activ- ities. Despite the great progress, one critical limitation of current brain decoding research lies in the lack of generalization capability to unseen subjects. Prior work typically focuses on decoding brain activity of individuals based on the ob- servation that different subjects exhibit different brain activities, while it remains unclear whether brain decoding can be generalized to unseen subjects. This study aims answer this question. We first consolidate an image-fMRI dataset con- sisting of stimulus-image and fMRI-response pairs, involving 177 subjects in the noeiinv task of the Human Connectome Project (HCP). This dataset allows us to investigate the brain decoding performance with the increase of participants. We then present a learning paradigm that applies uniform processing across all subjects, instead of employing different network heads or tokenizers for individu- als in previous methods, so that we can accommodate large number of subjects to explore the generalization capability across different subjects. A series of ex- periments are conducted and we have the following findings. First, the network exhibits clear generalization capabilities with the increase of training subjects. Second, the generalization capability is common to popular network architectures (MLP, CNN and Transformer). Third, the generalization performance is affected by the similarity between subjects. Our findings reveal the inherent similarities brain activities across individuals. With the emergence larger and more compre- hensive datasets, it is possible to train a brain decoding foundation model in the fu- ture. Codes and models can be found at
INTRODUCTION
Visual brain decoding (Kay et al.) 2008 Kamitani & Tong 2005
Naselaris et al., 2011 aims to
decode visual information from human brain activities, including tasks of brain- image classification (Kaur & Gandhi 2019, Zhou et al. 2024), retrieval (Scotti et al. 2024a Xia et al. 2024)) and recon- struction (Takagi & Nishimoto 2023] Ozcelik
2023)Ozcelik & VanRullen 2023 Ferrante et al. 2024 Scotti et al.)
2024a), and so It involves analyzing neural patterns collected brain imaging techniques like functional magnetic resonance imaging (fMRI) (Schirrmeister et al. 2017 Benchetrit et al. 2023]
Kamitani & Toog 2005) electroencephalograhyy (EEG) (Schirrmeister et al. [017]] Vallabha- neni et al. 2021) to infer the visual information received by the participants. Among them, fMRI favored by researchers because of its high spatial resolution and more informative depiction of the whole brain activity, which has resulted in a number of important decoding studies (Allen et al.)
2022
Takagi & Nishimoto 2023 Scotti et al. 2024a) with the help of deep learning techniques.
A major limitation of current brain decoding research, however, lies in the lack of generalization capability unseen subjects. That is, the trained decoding models can hardly be applied to new, unseen individuals not in the original training. Such a limitation can be owed to two reasons. First, there are individual differences of the brain activities across subjects (Haxby et al., 2020p. There- fore, it is assumed that brain decoding cannot be generalized and hence researchers are focused on
"Equal contributions. Corresponning authors.

---

Published as a conference paper at ICLR 2025
50%
TOP1 Accuracy
40%
45%
30%
37%
29%
20%
1.%
10%
2.%
0%
2
20
50
100
167
The Number of Training Subjects
Figure 1: The performance on unseen subjects with the increase of the number of training subjects.
developing subject-specific models. Second, commonly used brain visual decoding datasets are built upon only a small number of participants. For example, the Natural Scenes Dataset (NSD) dataset (Allen et al. (022) includes only 8 subjects. Most NSD-bbaeed studies (Kaur & Gandhi 2019 Scotti al. 20244) employ only 4 of the 8 subjects, and use the NSDGeneral data, which contain only the manually mapped brain regions, rather than the entire brain data (See more detailed discussions in Sec. B.1). Even those studies attempting to leverage multiple subjects are typically limited to less than I0 participants, and their networks are designed to handle only a small number of individuals. For instance, MindEye2 (Scotti al. (024b) and UMBRAE (Xia et al., 2024) use separate heads or tokenizers for different subjects. Therefore, the model becomes increasingly more complex with the increase of number of subject, which is hard to scale up to larger number of subjects.
In this work, we aim to address this limitation and answer the question of whether brain decoding can be generalized to unseen subjects. (The term "generalization" refers to the model's ability to accurately decode information from unseen subjects, rather than merely performing inference.) To this end, we first consolidated an image-fMRR dataset, which consists of pairs of the stimulus image and the corresponding brain fMRI response. We build this dataset using the data from the Human Connectome Project (HCP) (aan Essen al. 2013), which contains human brain neuroimages for various tasks. Among them, 177 subjects participated in the novie-viewing task, which provides the largest number of subjects available for extracting image- pairs for visual decoding study. In total, we collected 3,127 data pairs from 4 films watched by the 177 subjects. Compared to the commonly used datasets like NSD (8 subjects) (Allen et al., 2022) and BOLDS000 (4 subjects) (Chang et al. 2019), this dataset enables us to explore brain decoding performance with much larger number of subjects. We consequently propose a new learning paradigm. Following MindEyel (Scotti al. 2024a), we use CLIP encode the images, and employ brain decoding network to map brain activities (characterized by fMRI voxels) into the same CLIP space by contrastive learning. To handle the varying IMRI voxel sizes across subjects, we normalize them to a common size through upsampling. Unlike previous methods that rely on specially designed input heads or subject-specific tokenizers, our paradigm uses the same processing for all subjects so that it can handle a large number of subjects without increasing the model complexity and parameters. (See the example in Appendix.)
We perform experiments the fundamental retrieval task, which reflects well the capabilities of decoding models. Our method also has the potential to extend to reconstruction or grounding tasks with additional modules such as Stable Diffusion (Rombach et al. (022). Through detailed exper- imentation, we uncover several important and intriguing findings. First, as shown in Fig. [ the network demonstrates clear generalization ability as the number of training subjects increases, with top-1 accuracy rising from 2% (1 training subject) to 45% (167 training subjects) on unseen subjects (100 image-fMRI pairs). The accuracy can be further improved to 50% with additional training strategies. Second, the generalization capability holds for different network architectures. Using MLP, CNN and Transformer the backbone, we achieve top-1 accuracies of 45% 42%, and 34%, respectively, with 167 training subjects. Third, the generalization performance is influenced by sub- ject similarity. We observe bias when training on distinct groups such as gender, which represents one of the most casily identifiable categories of sample similarity. The model trained on 50 males achieves 36% top-1 accuracy on an unseen male subject, while the model trained on 50 females only obtains 27% top-1 accuracy on this test. Therefore, to explore further, we design an algorithm to calculate the similarity most similar and 20 least similar subjects. The models achieve 21 % and 2% top-1 accuracy, respec- tively, on an unseen subject, indicating the degree of similarity across subjects greatly affects the generalization performance. Our findings reveal that human brain activities share similarities, that require further exploration for generalization studies. It may be possible to train a large foundation model for brain decoding as bigger and more comprehensive datasets become available.
IMRI responses among 167 individuals and train two models on the 20
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
With the advancement of deep learning (Radford et al., 2021 He et al. 2016 Vaswani et al. 2017 Rombach al. 2022) and the emergence of high-quality fMRI datasets (Allen et al. [022] Chang et al.] 2019 Van Essen et al. 2013), many brain decoding methods with
Visual Brain Decoding.
promising performance have been proposed. [Tkagii & Nishimoto (2023) utilized latent diffusion model, specifically Stable Diffusion (Ho et al.) 2020 SSohl-Dickstein et al. 2015), to reconstruct high-resolution images from fMRI data, preserving semantic fidelity without requiring additional training or fine-tuning. Brain-Diffuser (Ozcelik & VanRullen 2023) improves the reconstruction process by first reconstructing basic image properties from fMRI signals and then refining the im- ages using latent diffusion model conditioned on multimodal features. Moreover, MindEye (Scotti) jetal.) 2024a) encodes images using CLIP and then maps the corresponding fMRI data the CLIP feature space, enabling strong image retrieval or reconstruction performance. However, most ex- isting methods focus on decoding stimuli for individual subjects. While effective for individual decoding, they lack the generalization capability to new, unseen subjects.
Visual Brain Decoding on Multiple Subjects. Some brain decoding methods have been devel- oped to leverage multiple subjects, which can be divided into two categories based on their objec- tives: (1) using multiple subjects to enhance subject-specific models, and (2) developing models that handle multiple subjects directly. For the first category, straightforward way to pre-train models multiple subjects and then fine-tune them for individual subjects (Scotti et al., 2024bb Jiang et al. 2024 Qian et al., 2023; Ferrante et al. 2024). For example, MindEEy2 (Scotti et al., 2024b) pre-trains the model on 7 subjects from the NSD dataset (Allen et al., 2022) and fine-tunes it on a different subject, using only 1/40 of the original data while achieving similar performance. The second category of methods aim to train model with multiple subjects that its performance on each subject (included in the training set) surpasses the models trained on each single subject. CLIP-MUSED (Zhou et al. 2024) and UMBRAE (Xia et al.. 2024) are methods of this kind. How- ever, most methods in this category still require separate heads or tokenizers for each subject. As the number of subjects increases, their training costs and model parameters grow linearly, making this approach impractical for larger subject pools. (We show an example in Appendix.) These multi-subject methods generally involve a limited number of subjects (less than 10), which are not sufficient enough for exploration. More importantly, while these methods demonstrate that certain information can be shared across subjects, they cannot generalize to unseen subjects.
Subjects Alignment. Some methods have been proposed to align new subjects to pre-trained mod- els, known as subject alignment, to handle unseen subjects. Based on the alignment approach, these methods can be categorized into anatomical alignment (Jenkinson et al. 2002) and functional align- ment (Haxby et al. 2011] Lorbert & Ramadge, 2012] Xu et al. 2012 Chen et al. 2015). among others. In visual brain decoding. the mainstream methods fall into functional alignment, which di- rectly aligns the neural activity patterns across different subjects. For instance, |Ferrante et al. (2024) used 1, 1,000 common images viewed by 8 subjects from the NSD dataset to train an alignment model that maps other subjects to Subject 1. During inference, the brain signals of other subjects are con- verted into the format of Subject and fed into the model trained on Subject 1. This approach can process new subjects with the model of existing subjects at lower cost, yet it requires shared data for alignment. In this work, we aim to achieve model generalization without such alignment.
METHODS
In this section, we first describe how we consolidate the dataset for exploring generalizable visual brain decoding in Sec. 3.1] Then, we describe the proposed learning paradigm in Sec. 3..2 Finally, in Sec. 3.3 we outline how we calculate the subject similarity.
3.1 DATASET CONSOLIDATION
Most previous studies (Scotti al. 2024a b Xia al. 2024 Zhou et al. 2024) are conducted on datasets with fewer than 10 participants, which cannot be used to study whether visual decoding can be generalizable. Therefore, to explore the generalization capabilities of brain decoding models, the first step is to collect a dataset with a larger number of subjects. However, as shown in Tab. T current publicly available image-viewing datasets are limited in size. For example, the NSD dataset
3

---

Published as a conference paper at ICLR 2025
Table 1: Summary of commonly used visual brain decoding datasets.
Dataset
Task
Scanner Subjects
Works Based on This Dataset
Chen et 2023 Prince al.
2022
BOLD5000 Chang et al.
2019
image-viewing
3T
4
Sexton & Love 2022
GOD
Horikuwa & Kamitani
2017
image- viegenn
3T
5
Chen et
2023 [ Du et al.
2023
MindEyel & Scotti al. 2024a
NSD
Allen al.
2022
image-viewing
7T
8
et. (2022
Ferrante
2024
Qian
al.
2023
Han et al.
2024
Raiders
laxby et al. 2011
movie-viewing
3T
Chen et
2015
Shvartsman
2018
Forrest Gump Hanke et al.
2014
movie-viewing
7T
20
Chen ct al
2015
Wagner ct al.
luang
[2022]
Budapest (Mateo et al.
2020
movie-viewing
3T
25
Matteo et al.
2021
Busch et
2021
HCP Van Essen ct al.
2013
movic-viewing
7T
177
Zhou ct al.
2024
Lu ct al
2024
Figure 2: Dataset reconstruc- tion from the HCP data. We extract the last frame i in each second of the movie clip as the stimulus image, and average the fMRI voxels in the subsequent 4 seconds (due to hemodynamic delay) the corresponding neural response to obtain image- [MRI pairs.
4
2
4
15
16
17
Lg
Scan
Movic Frames (Images)
12
V/
V/
F1
22
33
1/5
D7
V/S
Average for delay
IMRI Frames (Voxcls)
involves only 8 subjects (Allen et al. 2022 and BOLD5000 involves only 4 subjects (Chang et al,, 2019). This is mainly due to the high costs, time demands, and challenges in keeping participants engaged during the long fMRI scanning sessions. Furthermore, these existing datasets are hard combined due to their significant differences in scanning equipment, resolution, design and post-processing methods. Even if combined, the total number of subjects remains very small.
We then turn the movie-viewing task, which provides continuous, causally related visual inputs over short period. Compared to the image-viewing task, movie-viewing can yield more data pairs the same time-frame while keeping participants more engaged. Therefore, movie-viewing ex- periments involve more subjects, shown in Tab. [ Actually, some studies have proposed to extract video frames for visual brain retrieval (Schneider et al. (2023) and classification (Zhou et al.) 2024]. Therefore, we propose extract image-fMiK fMRI pairs from the movie-viewing task to build dataset. Specifically, we choose the movie-viewing task in the HCP dataset, which involves 177 par- ticipants, making it the largest movie-viewing dataset available for visual decoding research. In data collection, the participants watched four audiovisual films, and the IMRI responses were captured with repetition time (TR) of second using high-resolution 7T scanner. The volumetric images are registered to 1.6mm MNI space, with dimensions of 113 x 136 x 113 per TR.
As shown in Fig. of each second of the film as the stimulus image, whose corresponding fMRI response voxel is denoted as v.. Following the 4-senon hemodynamic delay suggested by Khosla et al. (2020), we average the fMRI signals from the subsequent 4 seconds to represent the neural response to each stimulus image (c.g., average oj vig to obtain V1 for i1), for total of 3,127 image-IMRI pairs for each subject. Finally, the reconstructed dataset includes 177 subjects with 177 x 3, 3,127 image-fMRI pairs. By leveraging this dataset, we can explore the generalization capabilities of brain decoding models across broader population. Our experiments in Sec. 4.2 demonstrate that generalization will emerge when a sufficient number of subjects are involved in training. This dataset provides a valuable foundation for researchers to investigate the behaviours of brain decoding models.
extract the corresponding image-fMRI pairs, we extract the last frame
4

---

Published as a conference paper at ICLR 2025
DODO
Suhj1
Subj2
Subj3
Subj4
Figure 3: The visualization of scanned brain data in NSD dataset. The highlighted regions indicate the manually labeled NSDGeneral data. Compared to the whole brain, the NSDGeneral regions show significant variations across different subjects.
3.2 LEARNING PARADIGM
Prior studies are typically focused on decoding brain activity of individuals, while little work has been done on exploring the model generalization capability to unseen subjects. With our consoli- dated dataset in Sec. [3.]] we propose a learning paradigm investigate the generalizability of visual brain decoding based on three core principles: (1) utilization of whole-brain data; (2) simple and flexible pipeline: (3) applicability to large number of diverse subjects.
Utilization of the Whole-Brain Data. As shown in Tab. [] most recent visual brain decoding studies rely on the NSD dataset, which provides two types of training data: NSDGeneral data and whole-brain data. As illustrated in Fig. 3 the thole-brain data contain the fMRI voxels (about 800K elements) of the entire brain, while the NSDGeneral data comprise 1D vectors (flattened voxels) of only 10k-20k elements, which are manually labeled as vision-related brain regions, called NSDGeneral regions (see the highlighted areas). Since NSDGeneral data are directly related to brain regions in charge visual processing, they often result in better visual decoding performance the studies focused single subjects. However, for research investigating the generalization capability across multiple subjects, we believe whole-brain data are more appropriate. First, the NSDGeneral data require manual segmentation, and hence they are difficult scale across large number of subjects, limiting their suitability for generalization studies. Note that most datasets, such as the HCP dataset, only provide whole-brain data. Second, as can be seen in Fig. 3 the manually labeled NSDGeneral regions show significant variations across different subjects. while the whole brain data (all the shown regions) show much less variation in shape. It is thus more difficult to train common model for multiple subjects using the NSDGeneral data than the whole-brain data (See Sec. (.5). Third, the NSDGeneral data exclude other brain regions, such as those in charge of memory, language, or contextual understanding. Ignoring those regions may prevent a more comprehensive decoding brain activities (Zhou et al. 2024 Xu et al., 202T). Therefore, we advocate for using whole-brain data in the study of on model generalization, rather than data limited to specific brain regions.
Simple and Flexible Pipeline. A simple and flexible learning pipeline is preferred to verify whether generalizability is a fundamental property of visual brain decoding, minimizing the fac- tors brought by complex network designs. Our learning pipeline is shown the left part of Fig- 4 The core idea is to project the paired stimulus-image I and iMRI-voxel V into the same feature space, where they could be as similar as possible. Following (Scotti et al.) 2024a] Xia et al. 2024), the CLIP ViT-L/14 model to encode the images into features Fi, while the visual brain de- coding network trained to map foRI-voxels to Fy in the same feature space. The feature size of the CLIP embedding space is 257 x 1024, which retains detailed image information compared to the high-level semantic content of the final CLS token in CLIP. Contrastive learning is employed to align F, with Fy using the CLIP Loss:
exp(sim Pj, P+)/T)
N
(fp (/)T))
Σ
log
log プ exp(sixpsmmm) (Ft, Ff)/T)
2N
N Σ
(FF()sim(FFFFF)))
(1)
5

---

Published as a conference paper at ICLR 2025
IMRI Voxels
Subject 11
Brain Decoding Network
Upsample
Contrastive Loss
Subject 177
Fy
MLP or 3D CNN MLP
Training Group 167 Subjects
Image Encoder (Such CLIP)
Backbone
(MLP, CNN or Transformer)
Subject 1-10
Testing Group 10 Unseen Subjects
MLP
fMRI Voxels V
Images
F
Figure 4: The overview of our learning pipeline (left) and visual brain decoding network (right).
where Fj and FV are the embeddings of the 7t image and IMRI voxel, the temperature param- eter, and sim(2, y) represents the cosine similarity between æ and
y
sim v) ||=|| |yyl
During inference, retrieval performed by calculating the cosine similarity and taking the most similar pairs. We use MLP the decoding network in most of our experiments, while the network architecture can changed to CNN and Transformer (See Sec. 4.3), and the model performance can be further improved with additional strategies (See Appendix).
Applicability to Many Diverse Subjects. Previous studies are mostly focused on a small number of participants. and they separate heads tokenizers for different subjects to improve per- formance. While being effective in small-scale studies, these approaches become impractical and cannot scale up the number of subjects increases. To explore model generalizability, we employ the same decoding network (see the right part of Fig. 4 to accommodate large number of subjects without requiring specific adaptations for each individual. Due the structural differences in brain anatomy, the size of fMRI voxels, even for the same brain activity, can vary across subjects, which cannot directly batched for network training. To solve this issue, we apply simple upsampling to resize the voxels to standardized larger size. This method is simple and straightforward and can be done in the processing the dataset. Experimental results show that this unified approach does not compromise performance in whole-brain decoding and can even enhance performance across mul- tiple subjects (See Sec. 4.5). As shown in the right of Fig. 4 our final network design involves an upsample layer to normalize voxel sizes for all subjects, followed by feeding the data into unified network without requiring any subject-specific adaptations.
3.3
GENERALIZATION PERFORMANCE VS. SUBJECT SIMILARITY
During experiments, notice performance biases of models trained different gender groups, which represent one of the most easily identifiable similarity categories. It inspires us to hypothesize that the degree similarity among subjects might impact the model generalization per- formance. To test this hypothesis, we need to identify which subjects are more similar to the given subjects. For target subject S,, given set of images viewed by N different subjects Sx, for each image i, we can calculate the cosine similarity (refer to Eq. 2 between the fMRI voxel Us,s, of target subject and the voxel v.ss, of subject S, E SN as follow: 3Sm__score,ss.
sim(vs,,, Vi,s.) ). The similarity score reflects the likeness between the two subjects (S, and S,) based on given image i. The overall similarity score of the two subjects can be obtained by averaging over all im- ages I. However, the outlier images make the averaged score less robust. Therefore, rank-based method. For each image i, we calculate Sim score, and rank the N scores from highest to lowest. Then, select the top 10 subjects based their ranks and award them one

---

Published as a conference paper at ICLR 2025
Table Results of models trained on our consolidated HCP dataset with different number training subjects. TOPI Acc. and TOP3 Acc. are averaged over the unseen subjects (Subjs 1-10). TOP1 Acc. (seen) is averaged over all seen subjects participated in training.
Training Subjects No. of Training Subjects TOPI Acc.
TOP3 Acc.
TOPI Acc. (seen)
Subj 11
2%
5%
79%
Subjs 11-12
2%
6%
84%
Subjs 11-30
20
15%
29%
83%
Subjs 1-60
50
29%
43%
83%
Subjs 11-110
100
37%
52%
82%
Subjs 11-177
167
45%
61%
82%
rank credit. After repeating this process for all images, subjects with higher total rank credits are considered more similar to the target subject St. The process to calculate the rank credit of Sn for S, can be formulated as:
Conk,Crrdii(S,.S... = 1SS, 1pp_11..ann(Smmscorree..
for j
N)), (3)
where 1-) is an indication function that assigns 1 if Sn, is among the top 10 most similar subjects to Sy based on image i, otherwise Finally, we use models trained on both similar and dissimilar subjects to explore how subject similarity influences the generalization performance. The results shown in Sec. 4.4
EXPERIMENTS
4.1 IMPLEMENTATION DETAILS
We implement all models using PyTorch (Paszke al. 2017). Except specifically indicated, employ MLP and 3D CNN as the backbone for feature extraction when using whole-brain data. The detailed network structure can be found in Appendix. During training, we employ the CLIP loss (Radford et al. 202T) and the AdamW optimizer (Loshchilov & Hutter 2017) to optimize the models (31 0.9, 32 = 0.999). We set the batch size 300, and apply the OneCycleLR strategy with a warm-up up phase to adjust the learning rate, with maximum learning rate of 1 x 10 The HCP dataset we consolidated includes 177 subjects, each subject having 3,127 image-IMRI pairs. We randomly choose 100 images and the corresponding fMRI voxels as the test pairs, and use the rest as the training pairs. Note that the test pairs of all subjects are from the same 100 images. Subjs 1-10 are designated as unseen subjects, with the remaining 167 subjects as seen subjects.
In our experiments, several models will be trained on different numbers of subjects. For convenience of expression, we define one training epoch based on the number of image-fMRI pairs of a single subject; that is, one epoch contains 3,027 image-fMil IMRI pairs. The numbers epuchs to train models on 1, 2, 20, 50, 100, and 167 seen subjects are 200, 200, 400, 600, 800, and 1,000, respectively. For the experiment on the NSD) dataset, which includes subjects, we follow the standard train/test split with 1,000 test images (Allen et al. 2022), and select Subj and Subj 5 unseen subjects. We train the models with and 6 seen subjects for 120 and 360 epochs, respectively. and each epoch includes 9,000 image-IMRI pairs.
4.2 MAIN RESULTS ON GENERALIZATION PERFORMANCE
As describe in Sec. 4.1 we train models 1, 2, 20, 50, 100 and 167 subjects and evaluate them on unseen subjects (Subjs 1-10). The results are shown in Tab. 2 We can clearly see that as the number of training subjects increases, the model's generalization capability unseen subjects improves. When only one or two subjects are used in training. the generalization capability is weak. When the number of training subjects reaches 167, the TOP1 and TOP3 accuracies improve to 45% and 61% %, respectively. (TOPI and TOP3 accuracies indicate the probability that the correct sample included among the top 1 or top 3 most relevant retrieval results returned by the network.)

---

Published as a conference paper at ICLR 2025
Table Results of models with differ- ent backbones trained on 167 subjects.
Backbones
TOPI Ace.
TOP3 Acc.
TOPI Acc. (seen)
MLP
45%
61%
82%
TOP1 Acc. and TOP3 Acc. are aver- aged over ten unseen subjects (Subjs 1- 10). TOPI Acc. (seen) averaged over all seen subjects.
ID CNN
42%
58%
80%
3D CNN
40%
57%
80%
Transformer
34%
52%
77%
Considering that our test set contains 100 image-fMRI pairs, such a generalization performance is highly encouraging. Fig. plots the curve of TOP1 accuracy vs. the number of training subjects. We observe that the generalization performance continues to improve steadily. Even with 167 subjects, it does not reach plateau. This suggests that the models hold potential for further improvement more subjects can be introduced for training.
We also provide in Tab. the results of our model on seen subjects for reference. The test pairs are from the subjects involved in the training process. We can see that the TOP1 accuracies are very close for different number of training subjects This is reasonable because the testing data share similar distribution with the training data, regardless of whether 1 or 167 subjects are involved, and the network able to fit the distribution via sufficient training.
4.3 THE GENERALIZATION PERFORMANCE WITH DIFFERENT BACKBONES
In Sec. we used MI.P as the backbone and validated the generalization capability of our models trained more subjects. Here, we validate whether this conclusion holds for other popular network architectures, such as CNN and Transformer networks. The details of the employed network archi- tectures can be found in Appendix. The results are shown in Tab. 3 which demonstrates that while there are some differences in performance, the generalization capability is consistently achieved across different network architectures. Specifically, the MLP achieves the best generalization per- formance, with 45% TOP1 accuracy on unseen subjects and 82% on seen subjects, followed closely by ID CNN and 3D CNN, which yield comparable results. The Transformer network exhibits the lowest performance, which may be attributed to the fact that Transformer typically needs larger training datasets to exhibit its superiority, whereas our current dataset for visual brain decoding relatively small, making CNNs and MLPs more effective in this case.
4.4 GENERALIZATION VS. SUBJECT SIMILARITY
From the experiments in previous sections, we have seen that the network could exhibit obvious generalization capability when enough subjects are used in training. During our experiments, inter- estingly, we notice some performance biases of models trained on different gender groups. Gender one of the most commonly observed characteristics of sample similarity, suggesting that the sim- ilarity among subjects might impact the model generalization performance. To be specific, three models using data from 50 male subjects, 50 female subjects, and a mixed group of 25 male and 25 female subjects, respectively. Then, evaluate these models unseen male Subj 1 and female Subj 2, and the results are shown in Tab. 4 One can see that the model trained on male sub- jects achieves the best retrieval performance on the unseen male subject (Subj 1) with 36% TOP1 accuracy and 60% TOP3 accuracy, but it performs the worst on the unseen female subject (Subj 2) female sub-
train
with 25% TOP1 accuracy and 37% TOP3 accuracy. In contrast, the model trained jects shows the opposite behaviour, performing better on Subj 2 and worse on Subj 1. Meanwhile, the model trained on the mixed group always obtain the intermediate result on the unseen subjects. Such results suggest that the generalization capability is related to the similarity among subjects.
Therefore, we further explore this phenomenon by finding similar and dissimilar subjects to Subj 1 and Subj 2. Utilizing the method described in Sec. 3.3 we identify 20 most similar subjects and 20 least similar subjects to Subj 1, as well as 20 most similar and least similar subjects to Subj 2. As shown in Tab. 4. the model trained the 20 subjects most similar to Subj achieves the best performance on Subj 1, with a TOP1 accuracy of 21% and a TOP3 accuracy of 36%. In contrast, the model trained on the 20 most dissimilar subjects performs significantly worse, with TOPI accuracy of 8% and a TOP3 accuracy of 14%. A similar trend can be observed for Subj 2, where the model trained on the 20 most similar dissimilar subjects achieves the highest lowest
8

---

Published as a conference paper at ICLR 2025
Table 4: Results of models trained on subjects that have different gender or similarity. All the training subjects are from Subj 3-167. Best and worst results are marked by red and blue.
TOPI Acc. on TOP3 Acc. on TOPI Acc. on TOP3 Acc. on Subj 1 (male) Subj 1 (male) Subj (female) Subj (female)
Training Subjects
50 male
36%
60%
25%
37%
50 female
27%
40%
30%
42%
25 male + 25 female
32%
49%
27%
40%
20 similar to Subj
21%
36%
20 dissimilar Subj
8%
14%
20 similar to Subj
21%
31%
20 dissimilar Subj
2%
7%
Subjs 11-30 (20)
17%
31%
18%
23%
20 similar 20 dissimilar Subj
21%
30%
20 similar + 20 dissimilar to Subj 2
20%
28%
Table 5: Results of models trained on NSD dataset. As in previous works, we randomly extracted 300 pairs from 1,000 pairs of the test set to perform the testing, and run 30 times the experiments to take the average. 'MindEyel' means our implemented model MindEyel (Scotti et al.
2024a).
NSDGeneral Data
NSD Whole-brain Data
Models
Training Subjects
Testing Subjects
Data Format
TOPI Acc.
Data Format TOPI Acc.
MindEyel
Subj 1
Subj
1 15,724
85%
83 x 104 81
35%
Ours
Subj 1
Subj I
x 18,000
86%
113 x 136 113
46%
MindEyel
Subj 7
Subj 7
1 12,682
70%
81 95 78
23%
Ours
Subj 7
Subj 7
18,000
70%
113 136 113
29%
Ours
Subjs 1,3,4,6,7,8
Subj 1
1 18,000
83%
113 x 136 113
49%
Ours
Subjs
1,3,4,6,7,8
Subj 7
x 18,000
69%
113 x 136 x 113
35%
Ours
Subjs 1,3,4,6,7,8
Subjs 2,5
18,000
1%
113 x 136 113
1%
Additionally, the model trained on Subjs 11-30, as reference for randomly selected 20 subjects, yields moderate performance similarity between subjects can largely affect the generalization performance. Even with 20 subjects in training, if the subjects are not similar, the model can achieve little generalization capability, such as 2% TOPI Acc. on Subj 2. We also train models on a mixed set of 20 similar and 20 dissimilar subjects for Subj 1 and Subj 2. The results closely match the performance of models trained on the 20 similar subjects alone.
performance.
both Subj 1 and Subj 2. This demonstrates that the
The above experimental results show that when the subjects similar, the models achieve better generalization performance, and vice versa. On the other hand, when a mix of similar and dissimilar subjects are used for training, generalization remains stable, with performance approaching the models trained on similar subjects. This suggests that generalization capability depends on learning inherent commonalities among human brains, with substantial tolerance for dissimilarities. It also explains why increasing the number of subjects enhances generalization - a larger dataset is more likely include subjects with higher similarities.
4.5 EXPERIMENTAL RESULTS ON THE NSD DATASET
To demonstrate the flexibility of our pipeline, we also train the model on the NSD dataset, including both NSDGeneral and whole-brain data. The results are shown in Tab. 5 We see that the models trained on Subj using NSDGeneral data with both the original data format (i.e., 1 x 15,724 in MindEyel) and our normalized data format (i.e., I 18,000) achieve similar TOP1 accuracy, i.e., 85% and 86%, respectively. (The TOPI accuracy reported in the original paper of MindEyel (Scotti et al. 2024a) is 84%.) The results Subj 7 can yield similar conclusion. This demonstrates that

---

Published as a conference paper at ICLR 2025
pipeline can be well applied to the individual -spectffassspeiiieeeeeeeeececeie scenario using NSDGeneral data. However, by using simple interpolation based upsampling to normalize the data format as a 1 x 18,000 vector, we can train models on multiple subjects with different original NSDGeneral data sizes. As shown in the bottom three rows of Tab. [}] by training on subjs 1,3,4,6,7.8 and testing on subj or subj 7, 83% and 69% TOP1 accuracy can still be obtained since subj subj 7 are included in the training data. However, when testing on the unseen subjs 2 and 5, only 1 % TOPI accuracy is obtained. This expected, as the NSD dataset consists of only 8 subjects, which is too limited ensure robust model generalization performance.
Finally, we evaluate the models trained with whole-brain data, The top right panel of Tab. 5 shows the results on Subj I and Subj 7 by MindEyel, which uses the original whole-brain data format, and our model, which uses the normalized data format (i.e., 113 x 136 x 113). We see that in the case of whole-brain data, the simple normalization of data size can improve much the TOP1 accuracy from 35% to 46% for subj and from 23% to 29% for subj 7. The accuracy is lower than that on NSDGeneral data because the NSDGeneral data are manually labeled brain visual regions. Again, the normalized data size enables us to train models on multiple subjects with different original data sizes, which cannot be done by MindEyel. As shown in the bottom right panel of Tab. [ our model trained on six subjects achieves 49% TOP1 accuracy on Subj and 35% on Subj 7, outperforming single-subject models trained on Subj (46%) and Subj 7 (29%), respectively. In contrast, on NSDGeneral data, the multiple-subject models perform slightly worse than their single- subject counterparts. This discrepancy highlights the seiiidual-speciffi nature of NSDGeneral data, as discussed in Sec. 3.1 (see also Fig. 3. We also report the generalization performance of models trained on 6 subjects on unseen Subjs 2 and 5. Again, the model shows weak generalization ability due to the small number of training subjects in NSD dataset.
5 DISCUSSION: THE SOURCE OF GENERALIZATION ABILITY
Based on the experimental results, we observe that when a sufficient number of subjects are included during training, various networks can exhibit generalization capabilities on unseen subjects. In this section, we discuss the potential causes underlying this generalization ability. Firstly, we assert that generalization does not arise from sophisticated upsampling (alignment) module or any other sim- ilar mechanism. Instead, we argue that it originates from "involving" rather than merely "aligning" the patterns across different subjects. In other words, the reason why our method performs well on unseen subjects is that the training data likely encompass the Saauuremmpppin patterns of unseen subjects. This hypothesis is supported by our following experimental findings. (1) Increasing Train- ing Subjects in in Sec. 4.2; As the number of training subjects increases, generalization improves. is likely because that the training set tends to include representative patterns of unseen subjects. (2) Subject Similarity in Sec. 4..4 When the number of training subjects is fixed, training groups that share greater similarity with the testing subjects (e.g., in terms of gender) tend to result in better gen- eralization performance. Our main contribution is to highlight this novel perspective: generalization ability can be obtained from "involving" a sufficiently diverse set of subjects, rather than relying on alignment or fine-tuning mechanisms; and generalization also depends on subject similarities.
6 CONCLUSION
Previous visual brain decoding studies typically focus on individual subjects, or training with multiple-subjects but decoding on seen subjects, while little work has been done to explore the possibility of generalizing visual brain decoding on unseen subjects. We attempt to fill this gap by leveraging a large dataset from the Human Connectome Project (HCP), constructing 177 x 3.127 image-IMRI pairs from 177 subjects. Using this dataset, we proposed learning paradigm, which utilized whole-brain data and a simple and uniform pipeline for processing all subjects, without requiring individual-specific adaptations. Via detailed experiments, we found that the model gener- alization capability emerged with the increase of training subjects, and such generalization capability hold across different network architectures. In addition, the similarity between subjects also played a role in improving the generalization capability. These findings reveal the inherent similarity in brain activities across individuals, which has significant implications for future studies. As larger and more diverse datasets may become available, this work can provide a basis for training a brain encoding foundation model for the future.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Emily J Allen, Ghislain St- Yves, Yihan Wu, Jesse L. Breedlove, Jacob S Prince, Logan T Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, et massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence. Nature neuroscience, 25(1): 25(1):116-126, 16-126, 2022.
Yohann Benchetrit, Hubert Banville, and Jean-Rémi King. Brain decoding: toward real-time recon- struction of visual perception. arXiv preprint arX1v:2310.19812, 2023.
Erica L Busch, Lukas Slipski, Ma Feilong, Swaroop Guntupalli, Matteo Visconti di Oleg- gio Castello, Jeremy F Huckins, Samuel A Nastase, M Ida Gobbini, Tor D Wager, and James V Haxby. Hybrid hyperalignment: single high- high-dimensional model of shared information em- bedded in cortical patterns of response and functional connectivity. Neurolmage, 233:117975, 2021.
Nadine Chang, John A Pyles, Austin Marcus, Abhinav Gupta, Michael Tarr, and Elissa M Aminoff. Bold5000, a public fmri dataset while viewing 5000 visual images. Scientific data, 6(1):49 2019.
Po-Hsuan Cameron Chen, Janice Chen, Yaara Yeshurun, Uri Hasson, James Haxby, and Peter J Ramadge. A veduced-dimensio finri shared response model. Advances in neural information processing systems, 28, 2015.
Zijiao Chen, Jiaxin Qing, Tiange Xiang, Wan Lin Yue, and Juan Helen Zhou. Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22710- 22720, 2023.
Changde Du Kaicheng Fu, Jinpeng Li, and Huiguang He. Decoding visual neural representations by multimodal learning of seiin-viuual--nnneee features. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9):10760-100 10777, 2023.
Matteo Ferrante, Tommaso Boccato, Furkan Ozcelik, Rufin VanRullen, and Nicola Toschi. Through their eyes: multi-subject brain decoding with simple alignment techniques. Imaging Neuro- science, 2:1-21, 2024.
Zijin Gu, Keith Jamison, Amy Kuceyeski, and Mert Sabuneu. Decoding natural image stimuli from fmri data with surface-based convolutional network. arXiv preprint arXiv:2212.02440, 2022.
Inhwa Han, Jaayeon Lee, and Jong Chul Ye. Mindformer: A transformer architecture for multi- subject brain decoding via fmri. arXiv preprint arXiv:2405.17720 2024.
Michael Hanke, Florian Baumgartner, Pierre Ibe, Falko R Kaule, Stefan Pollmann, Oliver Speck, Wolf Zinke, and Jörg Stadler. A high-resolution 7-tesla fmri dataset from complex natural stimu- lation with an audio movie. Scientific data, 1(1):1-18, 2014.
James V Haxby, J Swaroop Guntupalli, Andrew C Connolly, Yaroslav O Halchenko, Bryan R Con- roy, M Ida Gobbini, Michael Hanke, and Peter J Ramadge. A common, nigh-dimensional model of the representational space in human ventral temporal cortex. Neuron. 72(2):404-416 2011.
James V Haxby, Swaroop Guntupalli, Samuel A Nastase, and Ma Feilong. Hyperalignment: Modeling shared information encoded in idiosyncratic cortical topographies. elife, 9:e56601, 2020.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2016.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-663 0801, 2020.
Tomoyasu Horikawa and Yukiyasu Kamitani. Generic decoding of seen and imagined objects using hierarchical visual features. Nature communications, 851):15037, 2017.

---

Published as a conference paper at ICLR 2025
Jessie Huang, Erica Busch, Tom Wallenstein, Michal Gerasimiuk, Andrew Benz, Guillaume Lajoie, and Smita Krishnaswamy. Learning shared neural manifolds
Guy Wolf, Nicholas from multi-subject fmnri data. In 2022 IEEE 32nd International Workshop on Machine Learning for Signal Processing (MISP), pp. 01-06. IEEE, 2022.
Mark Jenkinson, Peter Bannister, Michael Brady, and Stephen Smith. Improved optimization for the robust and accurate linear registration and motion correction of brain images. Neuroimage, 17(2): 825-841 2002.
Shuai Jiang, Zhu Meng, Delong Liu, Haiwen Li, Fei Su, and Zhicheng Zhao. Mindshot: Brain decoding framework using only image. arXiv preprint arXiv:2405.15278 2024.
Yukiyasu Kamitani and Frank Tong. Decoding the visual and subjective contents of the human brain. Nature neuroscience, 8(5):679-685, 2005.
Taranjit Kaur and Tapan Kumar Gandhi. Automated brain image classification based on vgg- 16 and transfer learning. In 2019 international conference information technology (ICTT), pp. 94 98. IEEE, 2019.
Kendrick N Kay, Thomas Naselaris, Ryan Prenger. and Jack Gallant. Identifying natural images from human brain activity. Nature, 452(71855:352-335, 2008.
Meenakshi Khosla, Gia Ngo, Keith Jamison, Amy Kuceyeski, and Mert Sabuneu. A shared neural encoding model for the prediction of subject-specific fmri response. In Medical Image Computing and Computer Assisted Intervention-MICCAI 2020: 23rd International Conference, Lima, Peru, October 4-8, 2020, Proceedings, Part VII 23, pp. 539-548. Springer, 2020.
Alexander Lorbert and Peter J Ramadge. Kernel hyperalignment. Advances in Neural Information Processing Systems, 25, 2012.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05 2017.
Yizhuo Lu, Changde Du, Chong Wang. Xuanliu Zhu, Liuyun Jiang, and Huiguang He. Animate your thoughts: Decoupled reconstruction of dynamic natural vision from slow brain activity. arXiv preprint arXiv::405.0m 03280, 2024.
Visconti di Oleggio Castello Matteo, Vassiki Chauhan, Guo Jiahui, and M Ida Gobbini. An fmri dataset in response to "the grand budapest hotel", socially-rich, naturalistic movie. Scientific Data, 7(1):383, 2020.
Visconti Oleggio Castello Matteo, James Haxby, and M Ida Gobbini. Shared neural codes for visual and semantic information about familiar faces in common representational space. Proceedings of the National Academy of Sciences, 1 (1445)ee211074411 2021.
Thomas Naselaris, Kendrick N Kay, Shinji Nishimoto, and Jack L Gallant. Encoding and decoding in fmri. Neuroimage, 56(2):400-410, 2011.
Furkan Ozcelik and Rufin VanRullen. Natural scene reconstruction from fmri signals using genera- tive latent diffusion. Scientific Reports, 13(1):15666, 2023.
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Jacob Prince, Ian Charest, Jan W Kurzawski, John A Pyles, Michael Tarr, and Kendrick N Kay. Improving the accuracy of single-trial fmni response estimates using gimsingle. Elife, 11:777599, 2022.
Xuelin Qian, Yun Wang, Jingyang Huo, Jianfeng Feng, and Yanwei Fu. firi-pte: A large-scale fmrr pretrained transformer encoder for multi-subject brain activity decoding. arXiv preprint arXiv:2311.00344 2023.
12

---

Published as a conference paper at ICLR 2025
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning Transferable Visual Models From Natural Language Supervision. arXiv e-prints, art, arXiv:2103 1it::2133.0000,, February 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Robin Tibor Schirmeisster, Jost Tobias Springenberg. Lukas Dominique Josef Fiederer, Martin Glasstetter, Katharina Eggensperger, Michael Tangermann, Frank Hutter, Wolfram Burgard, and Tonio Ball. Deep learning with convolutional neural networks for eeg decoding and visualization. Human brain mapping,
2017.
Steffen Schneider, Jin Hwa Lee, and Mackenzie Weygandt Mathis. Learnable latent embeddings for joint behavioural and neural analysis. Nature, 617(7960):3603668 2023.
Paul Scotti, Atmadeep Banerjee, Jimmie Goode, Stepan Shabalin, Alex Nguyen, Aidan Dempster, Nathalic Verlinde, Elad Yundler, David Weisberg, Kenneth Norman, al. Reconstructing the mind's eye: fmri-to-image with contrastive learning and diffusion priors. Advances in Neural Information Processing Systems, 36, 2024a.
Paul Scotti, Mihir Tripathy, Cesar Kadir Torrico Villanueva, Reese Kneeland, Tong Chen, Ashutosh Narang, Charan Santhirasegaran, Jonathan Xu, Thomas Naselaris, Kenneth A Norman, et al. Mindeye2: -uaject models enable fmri-to-image with 1 hour of data. arXiv preprint arXiv:2403. 2007, 2024b.
Nicholas Sexton and Bradley C Love. Reassessing hierarchical correspondences between brain and deep networks through direct interface. Science advances, 8(28)eabm2219, 2022.
Michael Shvartsman, Narayanan Sundaram, Mikio Aoi, Adam Charles, Theodore Willke, and Jonathan Cohen. Matrix-normal models for fmri analysis. In International conference on ar- tificial intelligence and statistics, pp. 1914-1923. PMLR, 2018.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing, pp. 2256-2265. PMLR, 2015.
Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14453-14463, 2023.
Ramesh Babu Vallabhaneni, Pankaj Sharma, Vinit Kumar, Vyom Kulshreshtha, Koya Jeevan Reddy, S Sclva Kumar, V Sandeep Kumar, and Surendra Kumar Bitra. Deep learning algorithms in ceg signal decoding application: a review. IEEE Access, 9: 9:125778-125786, 2021.
David C Van Essen, Stephen M Smith, Deanna M Barch, Timothy EJ Behrens, Essa Yacoub, Kamil Ugurbil, Wu-Minn HCP Consortium, et al. The wu-minn human connectome project: an overview. Neuroimage, 80:62-79, 2013.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural informa- tion processing systems, 30, 2017.
Adina S Wagner, Laura Waite. Malgorzata Wierzba, Felix Hoffstaedter, Alexander Q Waite, Ben- jamin Poldrack, Simon B Eickhoff, and Michael Hanke. Fairly big: A framework for computa- tionally reproducible processing of large-scale data. Scientific data, 9(1:800, 2022.
Weihao Xia, Raoul de Charette, Cengiz Öztireli, and Jing- Hao Xue. Umbrac: Unified multimodal decoding of brain signals. arXiv preprint arXiv:2404. 07202,, 2024.
Hao Xu, Alexander Lorbert, Peter J Ramadge, J Swaroop Guntupalli, and James V Haxby. Regular- ized hyperalignment of multi-set fmni data. In 2012 IEEE statistical signal processing workshop (SSP), pp. 229-232. IEFF, 2012.
13

---

Published as a conference paper at ICLR 2025
Min Xu, Duo Li, and Ping Li. Brain decoding in multiple languages: Can cross- Lross-language brain decoding work? Brain and language, 215:104922, 2021.
Qiongyi Zhou, Changde Du, Shengpei Wang, and Huiguang He. Clip-mused: Clip-guided multi-
subject visual neural information semantic decoding. arXiv preprint arXiv:2402.08994 2024.
14


Reviews:
[{"review_id": "LvH5g44B1B", "reviewer": "Reviewer_DnEE", "summary": "This paper presents a new brain decoding paradigm for zero-shot brain decoding. Multiple experimental design exposes that brain decoding models can be generalized from existing subjects to unseen subjects.The effect of inter-subject similarity, such as gender, on generalization ability is discussed for the first time.", "questions": "I don't really agree with your statement in lines 506-508 of Section 4.6 that the poor generalization of the method on the NSD dataset is due to the problem of too few subjects. It seems to me at the moment that the integrated new dataset is consistent across all visual stimuli viewed by all subjects. So even for unseen subjects, the model is trained with the images that unseen subjects 'see'. However, for the NSD dataset, the images seen by different subjects, although they are all COCO images, are not the same in the training set. Therefore, in the absence of any comparison of related methods, how can it be proved that the method proposed in this paper is better not due to the difference in dataset ?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper is well written and easy for the reader to read.\n2. The motivation for the zero-shot setting is interesting.\n3. The experimental design is very rich with multiple validations. Readers are inspired by explorations of the impact of gender.\n4. In a sense, integrating the new dataset breaks the routine of NSD data use and is contributory.", "weaknesses": "1. From the point of view of machine learning, the algorithms for brain decoding in this paper are basically existing methods. The use of downsampling to obtain voxels of uniform size across multiple subjects has been used, e.g. CLIP-MUSED [1] uses PCA downsampling to align different subjects, also without additional training. \n2. The methodology mentioned by the authors was not compared to any relevant studies quantitatively. The methods mentioned by the authors were not compared to any relevant studies, so it is difficult to comparatively measure whether the methods proposed by the authors are more effective or not. There are actually a subset of papers that have performed the appropriate anatomical alignment for zero-shot brain decoding[2, 3, 4]. Because of open source issues, a full reproduction may be a bit difficult, but small designs such as the use of **fsaverage** are easy to implement (fsaverage is already widely used in neuroscience). \n3. While the new integrated dataset makes sense, it seems to me that a neuroscience research task without algorithmic design requires a dataset of its own design. I think this is missing in this paper. And the decoding task is very simple, containing only the retrieval task. Although the authors mention that the method can be generalized to the reconstruction task, a glance at related papers on the same method makes it hard for me to be convinced (the same method using downsampling fails miserably on the reconstruction task).\n\nOverall I do not deny the contribution of this paper and I think the research ideas in this paper are very interesting and deserve to be published. The reason I am currently leaning towards rejection is that I am concerned that the paper may be more suited to brain science related journals or conferences than machine learning conferences like **ICLR**. Because the methods in this paper do not involve any representation learning methods or any algorithmic innovations, and the conclusions are more of a scientific experimental nature. \n\nI would consider changing the rating to a positive one if the authors could convince me very well.\n\n**Reference**\n1. Zhou, Qiongyi, et al. \"CLIP-MUSED: CLIP-Guided Multi-Subject Visual Neural Information Semantic Decoding.\" ICLR 2024.\n2. Thual, Alexis, et al. \"Aligning brain functions boosts the decoding of visual semantics in novel subjects.\" ArXiv 2023.\n3. Bao, Guangyin, et al. \"Wills Aligner: A Robust Multi-Subject Brain Representation Learner.\" Arxiv 2024.\n4. Qian, Xuelin, et al. \"fmri-pte: A large-scale fmri pretrained transformer encoder for multi-subject brain activity decoding.\" Arxiv 2023.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "FAqRRclrBa", "reviewer": "Reviewer_x87x", "summary": "The paper explores cross-subject visual brain decoding, specifically focusing on scenarios where no data from unseen subjects are included in the training model. The contributions are twofold: first, the authors propose an effective, straightforward approach to generate image-fMRI data at scale using the HCP video viewing dataset. Second, they introduce a decoding method based on CLIP encoding combined with cosine similarity decoding. Although the approach is admittedly simple, it is practical and well-justified.", "questions": "Please provide a detailed rationale explaining why adding more subjects is beneficial. Specifically, how does this approach address the alignment problem? A clear explanation would help clarify the method's impact on the cross-subject variability and functional alignment issues inherent to neuroAI.\n\nPlease compare the performance of the proposed method with at least two other state-of-the-art approaches. This comparison would help establish its relative effectiveness and efficiency, particularly in terms of computational costs, accuracy, and robustness across subjects.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The writing is clear, and the rationale is logical. The question of alignment across subjects, which the paper addresses, is timely and aligns with topics discussed at prominent venues.", "weaknesses": "The decoding methodology is likely the paper’s most relevant contribution to ICLR, though I have identified two significant limitations in this area.\n\nFirstly, the authors show that incorporating data from additional subjects in training improves the model's performance, interpreting this as a solution to the alignment problem in neuroAI. In this field, we frequently encounter issues with requiring identical input-output pairs across subjects. While the authors suggest that their method mitigates this problem, they do not explain how. Is it possible that the model simply learns shared noise inherent to the videos across subjects as the data pool increases? We know that functional alignment remains a challenge because brain responses vary at the voxel level, even if the encoded information is similar across the whole region. It is unclear how, if at all, the proposed method addresses this challenge.\n\nSecondly, the paper lacks a direct comparison with other state-of-the-art methods, which the authors partially review in the introduction. Could alternative methods achieve similar or even better performance? The authors suggest that other methods are computationally expensive in terms of parameters and training, yet they do not provide specific details. Quantitative comparisons and a more in-depth analysis of parameter costs are essential to substantiate this claim.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JTSzzUCnI5", "reviewer": "Reviewer_QQAs", "summary": "The paper explores the generalization properties of visual decoding models when trained and tested on different subjects. Different model architectures (3D ConvNet, 1D ConvNet, Transformer, MLP) were trained on subsets of a dataset containing fMRI brain activity data collected while subjects watched audio-visual movies. Models were trained to recover the seen video frame out of a batch of images. The use of an upsampling step made it possible to reuse a single model across all subjects (vs. subject-specific layer in previous work). Results show that model performance increases as more subjects are seen during training, and that distribution shift between training and test subjects can negatively affect performance.", "questions": "1. The use of a video dataset to define an image retrieval task is interesting, however the distribution of images obtained by sampling frames every 1 s might be biased in a way that influences the interpretation of the results. Given frames within a scene are likely significantly correlated, the level of difficulty of the retrieval task on the test set will vary a lot if examples are strictly sampled from different scenes or not. Were the test images sampled uniformly across the 3127 images? A helpful visualization might be a grid of all sampled test images.\n2. Similarly, if a frame $i_j$ from was selected to be part of the test set and the surrounding frames that are part of the same scene (e.g. $i_{j-1}$ and $i_{j+1}$, which could be very similar to $i_j$ if the visual stimulus changes slowly) end up in the training set, there is a strong potential for overfitting. Could this be the case here?\n3. Averaging the 4-s window after the target frame is likely suboptimal as this misses the second half of the peak response (occuring at 4-s according to the reference cited). Have the authors looked at different pooling strategies (e.g. averaging seconds 2 to 6)?\n4. How many parameters do the different model configurations have? How were the hyperparameters selected? For instance I find it surprising that such a large Transformer was used (24 layers according to Suppl. Figure 4).\n5. What upsampling methodology was used?\n6. Previous studies on the NSD dataset relied on precomputed beta values obtained through a GLM fitting procedure that help improve SNR. Can the authors confirm whether the “whole-brain” data they used is also betas or is it BOLD signals? This could also be a more likely explanation for the discrepancy between results on NSDGeneral and “Whole-brain” data in Table 6.\n7. How were the images of Suppl Figure 1 selected? Are these retrieval results representative of the other images in the test set?\n8. Other comments:\n- At line 39, Benchetrit et al. 2023 is cited as an example of fMRI visual decoding, however they used MEG.\n- In the last sentence of the conclusion, the authors probably meant to say “a brain decoding foundation model” (instead of “encoding”) as brain encoding would instead mean reconstructing brain activity from the stimuli.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* Originality: While the proposed methodology is similar to previous work, the paper studies a setting (generalization across multiple subjects) which has not been explored much before. The analysis of the impact of training data composition (gender bias and of train-test set similarity) is also interesting and original.\n* Quality: The paper is of good quality and presents relevant experiments to support the claims made by the authors.\n* Clarity: The paper is overall well and clearly written, and the problem setting is well introduced and supported.\n* Significance: The results are promising and motivate further scaling up of brain activity datasets.", "weaknesses": "1. The description of the dataset curation and test set sampling procedure lacks details (see Q1-2). As movie frames are likely correlated and do not cover the same semantic space as “standard” brain-image datasets (in which the images are selected to cover varied categories, e.g. NSD), it is unclear whether the models might in fact overfit to the specific scenes seen in the training set. This could have an impact on the absolute performance values reported in the paper.\n2. There is also missing information about the architectures (upsampling procedure, number of parameters, hyperparameter selection procedure; see Q4-5).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "udZHXVgbMf", "reviewer": "Reviewer_cDXj", "summary": "The paper aims to tackle the challenge of visual brain decoding with a focus on improving generalization across a broader population of subjects. The authors generate a new dataset based on the HCP dataset, where image-fMRI pairs are derived from the last frame of each second in a movie, with the fMRI signals averaged over a specific time window. To map brain signals to the CLIP embeddings of corresponding images, they employ a contrastive learning approach. The discrepancy in voxel counts among subjects is addressed through a straightforward upsampling technique. Furthermore, the authors propose a novel architecture designed to mitigate the exponentially increasing model size typically required by state-of-the-art methods when integrating additional subjects into the pipeline.", "questions": "In addition to addressing the previously mentioned weaknesses, the following questions should be clarified to strengthen the paper:\n\nAre there any validation checks that confirm whether the fMRI signals accurately reflect the intended neural responses to individual images?\n\nAre there specific characteristics of whole-brain data that introduce redundancy or noise, which might explain the superior results of the hand-picked voxel approach?\n\nIn lines 66-67, it is stated that “the model becomes increasingly complex as the number of subjects increases, making it challenging to scale to a larger population,” referring to state-of-the-art architectures. It would be valuable to include a comparison of model size (in terms of the number of parameters) between the proposed approach and existing architectures. Such a comparison would help highlight the scalability advantages of the proposed model more clearly.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 1, "presentation": 2, "contribution": 1, "strengths": "The paper aims to address two major challenges in the field of visual brain decoding: the scarcity of existing datasets and the potential scalability issues of current pipelines when expanding to a larger number of subjects. However, the evidence provided for the proposed method is insufficient to fully support its effectiveness in tackling these problems.", "weaknesses": "1) I have concerns regarding the dataset generation methodology. It appears that the fMRIs generated for each image may not accurately capture the brain's response to the specific features of those images. Specifically, the resulting fMRI signals seem to reflect an accumulation of activity from prior stimuli, rather than providing a clear, one-to-one representation of the neural response to individual image features. This raises questions about whether the fMRI data truly aligns with the visual stimuli as intended.\n\n2) The validity of the generated dataset is questionable, particularly given the notable performance disparity observed between the NSD dataset and the paper's dataset (as noted in Table 2, row 1, compared to Table 6, row 2). It is unclear why there is such a significant difference, raising concerns about potential biases or inconsistencies in the data generation process that may not accurately reflect the neural representation of the images.\n\n3) In line 485, the paper reports MindEye 1's top-1 accuracy as 84%. However, according to Scotti et al. (2024a), MindEye 1 achieved a top-1 accuracy of 93.1% for image retrieval and 90.1% for brain retrieval (Table 1, row 5). In contrast, Scotti et al. (2024b) present MindEye 1's results as an average across four individual models (Table 1, row 3). Therefore, despite the mention of \"our implementation\" in line 488 which is unclear, it appears that this approach does not surpass the current state-of-the-art performance.\n\n4) Lines 51-52 suggest that hand-picking voxels is not ideal for data collection, yet the results in Table 6 (columns 5 and 7) demonstrate the superiority of this approach. This discrepancy raises questions about the claim. It is plausible that 'whole-brain' data could contain a significant amount of redundancy or noise, as suggested by prior research (Lin et al. \"Redundancy and dependency in brain activities.\" Shared Visual Representations in Human & Machine Intelligence (2022).), which may explain the better performance of the hand-picked voxel strategy.\n\n5) Table 6, row 7, indicates extremely poor performance, raising doubts about the pipeline’s ability to generalize effectively. In contrast, Scotti et al. (2024b), Table 1, row 7, report significantly better results even with a subset of the data. To strengthen the evaluation, a clear comparison between the proposed approach and MindEye2 is essential, as it would help clarify the relative effectiveness of the proposed method.\n\n6) As the authors noted, previous studies have employed various methods to address the discrepancies in voxel counts across subjects, such as creating a common representational space (e.g., Scotti et al., 2024b). However, it is unclear how a simple upsampling approach effectively manages the differences in fMRI signals among subjects. More details are needed on the implementation of the upsampling process. Additionally, an in-depth analysis comparing upsampling to existing alignment methods (e.g., functional alignment, anatomical alignment, common representational space) is necessary to understand whether it can truly serve as a viable alternative.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)