Toward Generalizing Visual Brain Decoding to Unseen Subjects

## Abstract

Abstract Visual brain decoding aims to decode visual information from human brain activities. Despite the great progress, one critical limitation of current brain decoding research lies in the lack of generalization capability to unseen subjects. Prior works typically focus on decoding brain activity of individuals based on the observation that different subjects exhibit different brain activities, while it remains unclear whether brain decoding can be generalized to unseen subjects. This study aims to answer this question. We first consolidate an image-fMRI dataset consisting of stimulus-image and fMRI-response pairs, involving 177 subjects in the movie-viewing task of the Human Connectome Project (HCP). This dataset allows us to investigate the brain decoding performance with the increase of participants. We then present a learning paradigm that applies uniform processing across all subjects, instead of employing different network heads or tokenizers for individuals as in previous methods, which can accommodate a large number of subjects to explore the generalization capability across different subjects. A series of experiments are conducted and we have the following findings. First, the network exhibits clear generalization capabilities with the increase of training subjects. Second, the generalization capability is common to popular network architectures (MLP, CNN and Transformer). Third, the generalization performance is affected by the similarity between subjects. Our findings reveal the inherent similarities in brain activities across individuals. With the emerging of larger and more comprehensive datasets, it is possible to train a brain decoding foundation model in the future. Codes and models can be found at https://github.com/Xiangtaokong/TGBD .

## 1 Introduction

Visual brain decoding (Kay et al., ; Kamitani & Tong, ; Naselaris et al., ) aims to decode visual information from human brain activities, including tasks of brain-image classification (Kaur & Gandhi, ; Zhou et al., ), retrieval (Scotti et al., ; Xia et al., ) and reconstruction (Takagi & Nishimoto, ; Ozcelik & VanRullen, ; Ferrante et al., ; Scotti et al., ), and so on.
It involves analyzing neural patterns collected via brain imaging techniques like functional magnetic resonance imaging (fMRI) (Schirrmeister et al., ; Benchetrit et al., ; Kamitani & Tong, ) or electroencephalography (EEG) (Schirrmeister et al., ; Vallabhaneni et al., ) to infer the visual information received by the participants.
Among them, fMRI is favored by researchers because of its more informative depiction of the whole brain activity, which has resulted in a number of important decoding works (Allen et al., ; Takagi & Nishimoto, ; Scotti et al., ) with the help of deep learning techniques.

A major limitation of current brain decoding research, however, lies in the lack of generalization capability to unseen subjects.
That is, the trained decoding models can hardly be applied to new, unseen individuals.
Such a limitation can be owed to two reasons.
First, there are individual differences of the brain activities across subjects (Haxby et al., ).
Therefore, it is assumed that brain decoding cannot be generalized and hence researchers are focused on developing subject-specific models.
Second, commonly used brain visual decoding datasets are built upon only a small number of participants.
For example, the Natural Scenes Dataset (NSD) dataset (Allen et al., ) includes only 8 subjects.
Most NSD-based studies (Kaur & Gandhi, ; Scotti et al., ) employ only 4 of the 8 subjects, and use the NSDGeneral data, which contain only the manually mapped brain region, rather than the entire brain data (See more detailed discussions in Sec. 3.1).
Even those studies attempting to leverage multiple subjects are typically limited to less than 10 participants, and their networks are designed to handle only a small number of individuals.
For instance, MindEye2 (Scotti et al., ) and UMBRAE (Xia et al., ) use separate heads or tokenizers for different subjects. Therefore, the model becomes increasingly complex as the increase of subject number, which is hard to scale up to a larger number of subjects.

Figure: Figure 1: The performance on unseen subjects with the increase of the number of training subjects.
Refer to caption: x1.png

In this work, we aim to address this limitation and answer the question that whether brain decoding can be generalized to unseen subjects.
To this end, we first consolidated an image-fMRI dataset, which consists of pairs of the stimulus image and the corresponding brain fMRI response. We build this dataset using the data from the Human Connectome Project (HCP) (Van Essen et al., ), which contains human brain neuroimages for various tasks.
Among them, 177 subjects participated in the movie-viewing task, which provides the largest number of subjects available for extracting image-fMRI pairs for visual decoding study.
In total, we collected 3,127 data pairs from 4 films watched by the 177 subjects.
Compared to the commonly used datasets like NSD (8 subjects) (Allen et al., ) and BOLD5000 (4 subjects) (Chang et al., ), this dataset enables us to explore brain decoding performance with a much larger number of subjects.
We consequently propose a new learning paradigm.
Following MindEye1 (Scotti et al., ), we use CLIP to encode the images, and employ a brain decoding network to map brain activities (characterized by fMRI voxels) into the same CLIP space by contrastive learning.
To handle the varying fMRI voxel sizes across subjects, we simply normalize them to a common size through upsampling.
Unlike previous methods that rely on specially designed input heads or subject-specific tokenizers, our paradigm uses the same processing for all subjects so that it can handle a large number of subjects without increasing the model complexity and parameters.

We perform experiments on the fundamental retrieval task, which reflects well the capabilities of decoding models. (Our method can be extended to reconstruction or grounding tasks with additional modules such as Stable Diffusion (Rombach et al., ).)
Through detailed experimentation, we uncover several important and intriguing findings.
First, as shown in Fig. 1, the network demonstrates clear generalization ability as the number of training subjects increases, with top-1 accuracy rising from 2% (1 training subject) to 45% (167 training subjects) on unseen subjects (100 image-fMRI pairs). The accuracy can be further improved to 50% with additional training strategies.
Second, the generalization capability holds for different network architectures.
Using MLP, CNN and Transformer as the backbone, we achieve top-1 accuracies of 45%, 42%, and 34%, respectively, with 167 training subjects.
Third, the generalization performance is influenced by subject similarity.
We observe a bias when training on distinct groups such as gender, which represents one of the most easily identifiable categories of sample similarity.
The model trained on 50 males achieves 36% top-1 accuracy on an unseen male subject, while the model trained on 50 females only obtains 27% top-1 accuracy on this test.
Therefore, to explore further, we design an algorithm to calculate the similarity of fMRI responses among 167 individuals and train two models on the 20 most similar and 20 least similar subjects.
The models achieve 21% and 2% top-1 accuracy, respectively, on an unseen subject, indicating the degree of similarity across subjects greatly affects the generalization performance.
Our findings reveal that human brain activities share similarities, which is worth of further exploration. It may be possible to train a large foundation model for brain decoding as bigger and more comprehensive datasets emerge.

## 2 Related Work

#### Visual Brain Decoding.

With the advancement of deep learning (Radford et al., ; He et al., ; Vaswani et al., ; Rombach et al., ) and the emergence of high-quality fMRI datasets (Allen et al., ; Chang et al., ; Van Essen et al., ), many brain decoding methods with promising performance have been proposed.
Takagi & Nishimoto () utilized a latent diffusion model, specifically Stable Diffusion (Ho et al., ; Sohl-Dickstein et al., ), to reconstruct high-resolution images from fMRI data, preserving semantic fidelity without requiring additional training or fine-tuning.
Brain-Diffuser (Ozcelik & VanRullen, ) improves the reconstruction process by first reconstructing basic image properties from fMRI signals and then refining the images using a latent diffusion model conditioned on multimodal features.
Moreover, MindEye (Scotti et al., ) encodes images using CLIP and then maps the corresponding fMRI data to the CLIP feature space, enabling strong image retrieval or reconstruction performance.
However, most existing methods focus on decoding stimuli for individual subjects.
While effective for individual decoding, they lack the generalization capability to new, unseen subjects.

#### Visual Brain Decoding on Multiple Subjects.

Some brain decoding methods have been developed to leverage multiple subjects, which can be categorized into two categories based on their objectives: (1) using multiple subjects to enhance subject-specific models, and (2) developing models that handle multiple subjects directly.
For the first category, a straightforward way is to pre-train models on multiple subjects and then fine-tune them for individual subjects (Scotti et al., ; Jiang et al., ; Qian et al., ; Ferrante et al., ).
For example, MindEye2 (Scotti et al., ) pre-trains the model on 7 subjects from the NSD dataset (Allen et al., ) and fine-tunes it on a different subject, using only 1/40 of the original data while achieving similar performance.
The second category of methods aim to train a model with multiple subjects so that its performance on each subject (included in the training set) surpasses the models trained on each single subject.
CLIP-MUSED (Zhou et al., ) and UMBRAE (Xia et al., ) are methods of this kind.
However, most methods in this category still require separate heads or tokenizers for each subject. As the number of subjects increases, their training costs and model parameters grow linearly, making this approach impractical for larger subject pools.
These multi-subject methods generally involve a limited number of subjects (less than 10), which are not sufficient enough for exploration.
More importantly, while these methods demonstrate that certain information can be shared across subjects, they cannot generalize to unseen subjects.

#### Subjects Alignment.

Some methods have been proposed to align new subjects to pre-trained models, known as subject alignment, to handle unseen subjects.
Based on the alignment approach, these methods can be categorized into anatomical alignment (Jenkinson et al., ) and functional alignment (Haxby et al., ; Lorbert & Ramadge, ; Xu et al., ; Chen et al., ), among others.
In visual brain decoding, the mainstream methods fall into functional alignment, which directly aligns the neural activity patterns across different subjects.
For instance, Ferrante et al. () used 1,000 common images viewed by 8 subjects from the NSD dataset to train an alignment model that maps other subjects to Subject 1.
During inference, the brain signals of other subjects are converted into the format of Subject 1 and fed into the model trained on Subject 1.
This approach can process new subjects with the model of existing subjects at a lower cost, yet it requires shared data for alignment.
In this work, we aim to achieve model generalization without such alignment.

## 3 Methods

In this section, we first describe how we consolidate the dataset for exploring generalizable visual brain decoding in Sec. 3.1.
Then, we describe the proposed learning paradigm in Sec. 3.2.
Finally, in Sec. 3.3 we outline how we calculate the subject similarity.

### 3.1 Dataset Consolidation

**Table 1: Summary of commonly used visual brain decoding datasets.**
| Dataset | Task | Scanner | Subjects | Works Based on This Dataset |
| --- | --- | --- | --- | --- |
| BOLD5000 (Chang et al., ) | image-viewing | 3T | 4 | Chen et al. (); Prince et al. (); Sexton & Love () |
| \cdashline<br>1-5<br>GOD (Horikawa & Kamitani, ) | image-viewing | 3T | 5 | Chen et al. (); Du et al. () |
| \cdashline<br>1-5<br>NSD (Allen et al., ) | image-viewing | 7T | 8 | MindEye1&2 (Scotti et al., ; ); Gu et al. (); Ferrante et al. (); Qian et al. (); Han et al. () |
| Raiders (Haxby et al., ) | movie-viewing | 3T | 21 | Chen et al. (); Shvartsman et al. () |
| \cdashline<br>1-5<br>Forrest Gump (Hanke et al., ) | movie-viewing | 7T | 20 | Chen et al. (); Wagner et al. (); Huang et al. () |
| \cdashline<br>1-5<br>Budapest (Matteo et al., ) | movie-viewing | 3T | 25 | Matteo et al. (); Busch et al. () |
| \cdashline<br>1-5<br>HCP (Van Essen et al., ) | movie-viewing | 7T | 177 | Zhou et al. (); Lu et al. () |

Most previous studies (Scotti et al., ; ; Xia et al., ; Zhou et al., ) are conducted on datasets with fewer than ten participants, which cannot be used to study whether visual decoding can be generalizable.
Therefore, to explore the generalization capabilities of brain decoding models, the first step is to collect a dataset with a larger number of subjects.
However, as shown in Tab. 1, current publicly available image-viewing datasets are limited in size. For example, the NSD dataset involves only 8 subjects (Allen et al., ) and BOLD5000 involves only 4 subjects (Chang et al., ).
This is mainly due to the high costs, time demands, and challenges in keeping participants engaged during the long fMRI scanning sessions.
Furthermore, these existing datasets are hard to be combined due to their significant differences in scanning equipment, resolution, and post-processing methods.
Even combined, the total number of subjects remains very small.

We then turn to the movie-viewing task, which provides continuous, causally-related visual inputs over a short period.
Compared to the image-viewing task, movie-viewing can yield much more data pairs in the same time-frame while keeping participants more engaged. Therefore, movie-viewing experiments can involve more subjects, as shown in Tab. 1.
Actually, some works have been proposed to extract video frames for visual brain retrieval (Schneider et al., ) and classification (Zhou et al., ).
Therefore, we propose to extract image-fMRI pairs from the movie-viewing task to build our dataset.
Specifically, we choose the movie-viewing task in the HCP dataset, which involves 177 participants, making it the largest movie-viewing dataset available for visual decoding research.
In data collection, the participants watched four audiovisual films, and the fMRI responses were captured with a repetition time (TR) of 1 second using a high-resolution 7T scanner.
The volumetric images are registered to 1.6mm MNI space, with dimensions of 113 × 136 × 113 per TR.

As shown in Fig. 2, to extract the corresponding image-fMRI pairs, we extract the last frame $i$ of each second of the film as the stimulus image, whose corresponding fMRI response voxel is denoted as $v^{,}_{i}$.
Following the 4-second hemodynamic delay suggested by Khosla et al. (), we average the fMRI signals from the subsequent four seconds to represent the neural response to each stimulus image (*e.g.*, average $v^{,}_{1}$ - $v^{,}_{5}$ to obtain $v_{1}$ for $i_{1}$), resulting in a total of 3,127 image-fMRI pairs for each subject.
Finally, the reconstructed dataset includes 177 subjects with 177 × 3,127 image-fMRI pairs.
By leveraging this dataset, we can explore the generalization capabilities of brain decoding models across a broader population.
Our experiments in Sec. 4.2 demonstrate that generalization will emerge when a sufficient number of subjects are involved in training.
This dataset provides a valuable foundation for researchers to investigate the behaviours of brain decoding models.

Figure: Figure 2: Dataset reconstruction from the HCP data. We extract the last frame $i$ in each second of the movie clip as the stimulus image, and average the fMRI voxels in the subsequent 4 seconds (due to hemodynamic delay) as the corresponding neural response $v$ to obtain image-fMRI pairs.
Refer to caption: x2.png

### 3.2 Learning Paradigm

Prior studies are typically focused on decoding brain activity of individuals, while little work has been done on exploring the model generalization capability to unseen subjects.
With our consolidated dataset in Sec. 3.1, we propose a learning paradigm to investigate the generalizability of visual brain decoding based on three core principles:
(1) utilization of whole-brain data;
(2) simple and flexible pipeline;
(3) applicability to a large number of diverse subjects.

#### Utilization of the Whole-Brain Data.

Figure: Figure 3: The visualization of scanned brain data in NSD dataset. The highlighted regions indicate the manually labeled NSDGeneral data. Compared to the whole brain, the NSDGeneral regions show significant variations across different subjects.
Refer to caption: x3.png

As shown in Tab. 1, most recent visual brain decoding studies rely on the NSD dataset, which provides two types of training data: NSDGeneral data and whole-brain data.
As illustrated in Fig. 3, the whole-brain data contain the fMRI voxels (about 800K elements) of the entire brain, while the NSDGeneral data comprise 1D vectors (flattened voxels) of only 10k–20k elements, which are manually labeled as vision-related brain regions, called NSDGeneral regions (see the highlighted areas).
Since NSDGeneral data are directly related to brain regions in charge of visual processing, they often result in better visual decoding performance in the studies focused on single subjects.
However, for research investigating the generalization capability across multiple subjects, we believe whole-brain data are more appropriate.
First, the NSDGeneral data require manual segmentation, and hence they are difficult to scale across a large number of subjects, limiting their suitability for generalization studies.
Note that most datasets, such as the HCP dataset, only provide whole-brain data.
Second, as can be seen in Fig. 3, the manually labeled NSDGeneral regions show significant variations across different subjects, while the whole brain data show much less variations in shape.
It is thus more difficult to train a common model to multiple subjects using the NSDGeneral data than the whole-brain data (See Sec. 4.6).
Third, the NSDGeneral data exclude other brain regions, such as those in charge of memory or contextual understanding.
Ignoring those regions may prevent a more comprehensive decoding of brain activities (Zhou et al., ).
Therefore, we advocate for using whole-brain data in the study of on model generalization, rather than data limited to specific brain regions.

#### Simple and Flexible Pipeline.

A simple and flexible learning pipeline is preferred to verify whether generalizability is a fundamental property of visual brain decoding, minimizing the factors brought by complex network designs.
Our learning pipeline is shown in the left part of Fig. 4. The core idea is to project the paired stimulus-image $I$ and fMRI-voxel $V$ into the same feature space, where they could be as similar as possible.
Following (Scotti et al., ; Xia et al., ), we use the CLIP ViT-L/14 model to encode the images into features $F_{I}$, while the visual brain decoding network is trained to map fMRI-voxels to $F_{V}$ in the same feature space.
The feature size of the CLIP embedding space is $257\times 1024$, which retains detailed image information compared to the high-level semantic content of the final CLS token in CLIP.
Contrastive learning is employed to align $F_{I}$ with $F_{V}$ using the CLIP Loss:

$$ $\mathcal{L}=\frac{1}{2N}\left(\sum_{i=1}^{N}-\log\frac{\exp(\text{sim}(F_{I}^{ i},F_{V}^{i})/\tau)}{\sum_{j=1}^{N}\exp(\text{sim}(F_{I}^{i},F_{V}^{j})/\tau)} +\sum_{i=1}^{N}-\log\frac{\exp(\text{sim}(F_{V}^{i},F_{I}^{i})/\tau)}{\sum_{j= 1}^{N}\exp(\text{sim}(F_{V}^{i},F_{I}^{j})/\tau)}\right),$ (1) $$

where $F_{I}^{i}$ and $F_{V}^{i}$ are the embeddings of the $i^{th}$ image and fMRI voxel, $\tau$ is the temperature parameter, and $\text{sim}(x,y)$ represents the cosine similarity between $x$ and $y$:

$$ $\text{sim}(x,y)=\frac{x\cdot y}{\|x\|\|y\|}.$ (2) $$

During inference, retrieval is performed by calculating the cosine similarity and taking the most similar pairs.
We use MLP as the decoding network in most of our experiments, while the network architecture can be changed to CNN and Transformer (See Sec. 4.3), and the model performance can be further improved with additional strategies (See Sec. 4.5).

#### Applicability to Many Diverse Subjects.

Previous studies are mostly focused on a small number of participants, and they use separate heads or tokenizers for different subjects to improve performance.
While being effective in small scale studies, these approaches become impractical and cannot scale up as the number of subjects increases.
To explore model generalizability, we employ the same decoding network (see the right part of Fig. 4) to accommodate a large number of subjects without requiring specific adaptations for each individual.
Due to the structural differences in brain anatomy, the size of fMRI voxels, even for the same brain activity, can vary across subjects, which cannot be directly batched for network training.
To solve this issue, we apply simple upsampling to resize the voxels to a standardized larger size.
This method is simple and straightforward and can be done in the processing of the dataset.
Experimental results show that this unified approach does not compromise performance in whole-brain decoding and can even enhance performance across multiple subjects (See Sec. 4.6).
As shown in the right of Fig. 4, our final network design involves an upsample layer to normalize voxel sizes for all subjects, followed by feeding the data into a unified network without requiring any subject-specific adaptations.

Figure: Figure 4: The overview of our learning pipeline (left) and visual brain decoding network (right).
Refer to caption: x4.png

### 3.3 Generalization Performance vs. Subject Similarity

During experiments, we notice some performance biases of models trained on different gender groups, which represent one of the most easily identifiable similarity categories.
It inspires us to hypothesize that the degree of similarity among subjects might impact the model generalization performance.
To test this hypothesis, we need to identify which subjects are more similar to the given subjects.
For a target subject $S_{t}$, given a set of images $I$ viewed by $N$ different subjects $S_{N}$, for each image $i$, we can calculate the cosine similarity (refer to Eq. 2) between the fMRI voxel $v_{i,S_{t}}$ of target subject and the voxel $v_{i,S_{n}}$ of subject $S_{n}\in S_{N}$ as follow: $Sim_score_{i,S_{t},S_{n}}=sim(v_{i,S_{t}},v_{i,S_{n}})$.
The similarity score reflects the likeness between the two subjects ($S_{t}$ and $S_{n}$) based on a given image $i$. The overall similarity score of the two subjects can be obtained by averaging over all images $I$. However, the outlier images can make the averaged score less robust.
Therefore, we use a rank-based method. For each image $i$, we calculate $Sim_score_{i,S_{t},S_{n}}$ and rank the $N$ scores from highest to lowest.
Then, we select the top 10 subjects based on their ranks and award them one rank credit.
After repeating this process for all images, subjects with higher total rank credits are considered more similar to the target subject $S_{t}$.
The process to calculate the rank credit of $S_{n}$ for $S_{t}$ can be formulated as:

$$ $\text{Rank_Credit}(S_{t},S_{n})=\sum\nolimits_{i=1}^{I}\mathbbm{1}(S_{n}\in \text{top_10_rank}(Sim_score_{i,S_{t},S_{j}}\text{ for }j=1,2\ldots,N)),$ (3) $$

where $\mathbbm{1}(\cdot)$ is an indication function that assigns 1 if $S_{n}$ is among the top 10 most similar subjects to $S_{t}$ based on image $i$, otherwise 0.
Finally, we use models trained on both similar and dissimilar subjects to explore how subject similarity influences the generalization performance. The results are shown in Sec. 4.4.

## 4 Experiments

### 4.1 Implementation Details

We implement all models using PyTorch (Paszke et al., ).
Except specifically indicated, we employ MLP and 3D CNN as the backbone for feature extraction when using whole-brain data.
The detailed network structure can be found in the appendix.
During training, we employ the CLIP loss (Radford et al., ) and the AdamW optimizer (Loshchilov & Hutter, ) to optimize the models ($\beta_{1}$ = 0.9, $\beta_{2}$ = 0.999).
We set the batch size to 300, and apply the OneCycleLR strategy with a warm-up phase to adjust the learning rate, with a maximum learning rate of $1\times 10^{-4}$.
The HCP dataset we consolidated includes 177 subjects, each subject having 3,127 image-fMRI pairs.
We randomly choose 100 images and the corresponding fMRI voxels as the test pairs, and use the rest as the training pairs.
Note that the test pairs of all subjects are from the same 100 images.
Subjs 1-10 are designated as unseen subjects, with the remaining 167 subjects as seen subjects.

In our experiments, several models will be trained on different numbers of subjects. For convenience of expression, we define one training epoch based on the number of image-fMRI pairs of a single subject; that is, one epoch contains 3,027 image-fMRI pairs.
The numbers of epochs to train models on 1, 2, 20, 50, 100, and 167 seen subjects are 200, 200, 400, 600, 800, and 1,000, respectively.
For the experiment on the NSD dataset, which includes 8 subjects, we follow the standard train/test split with 1,000 test images (Allen et al., ), and select Subj 2 and Subj 5 as unseen subjects.
We train the models with 1 and 6 seen subjects for 120 and 360 epochs, respectively, and each epoch includes 9,000 image-fMRI pairs.

### 4.2 Main Results on Generalization Performance

**Table 2: Results of models trained on our consolidated HCP dataset with different number of training subjects. TOP1 Acc. and TOP3 Acc. are averaged over the unseen subjects (Subjs 1-10). TOP1 Acc. (seen) is averaged over all seen subjects participated in training.**
| Training Subjects | No. of Training Subjects | TOP1 Acc. | TOP3 Acc. | TOP1 Acc. (seen) |
| --- | --- | --- | --- | --- |
| Subj 11 | 1 | 2% | 5% | 79% |
| Subjs 11-12 | 2 | 2% | 6% | 84% |
| Subjs 11-30 | 20 | 15% | 29% | 83% |
| Subjs 11-60 | 50 | 29% | 43% | 83% |
| Subjs 11-110 | 100 | 37% | 52% | 82% |
| Subjs 11-177 | 167 | 45% | 61% | 82% |

As describe in Sec. 4.1, we train models on 1, 2, 20, 50, 100 and 167 subjects and evaluate them on 10 unseen subjects (Subjs 1-10).
The results are shown in Tab. 2. We can clearly see that as the number of training subjects increases, the model’s generalization capability on unseen subjects improves.
When only one or two subjects are used in training, the generalization capability is weak. When the number of training subjects reaches 167, the TOP1 and TOP3 accuracies improve to 45% and 61%, respectively.
Considering that our test set contains 100 image-fMRI pairs, such a generalization performance is highly encouraging.
Fig. 1 plots the curve of TOP1 accuracy vs. the number of training subjects. We observe that the generalization performance continues to improve steadily. Even with 167 subjects, it does not reach a plateau.
This suggests that the models hold potential for further improvement if more subjects can be introduced for training.

We also provide in Tab. 2 the results of our model on seen subjects for reference.
The test pairs are from the subjects involved in the training process. We can see that the TOP1 accuracies are very close for different number of training subjects.
This is reasonable because the testing data share similar distribution with the training data, no matter 1 or 167 subjects are involved, and the network is able to fit the distribution via sufficient training.

### 4.3 The Generalization Performance with Different Backbones

**Table 3: Results of models with different backbones trained on 167 subjects. TOP1 Acc. and TOP3 Acc. are averaged over ten unseen subjects (Subjs 1-10). TOP1 Acc. (seen) is averaged over all seen subjects.**
| Backbones | TOP1 Acc. | TOP3 Acc. | TOP1 Acc. (seen) |
| --- | --- | --- | --- |
| MLP | 45% | 61% | 82% |
| 1D CNN | 42% | 58% | 80% |
| 3D CNN | 40% | 57% | 80% |
| Transformer | 34% | 52% | 77% |

In Sec. 4.2, we used MLP as the backbone and validated the generalization capability of our models trained on more subjects. Here, we validate whether this conclusion holds for other popular network architectures, such as CNN and Transformer networks.
The details of the employed network architectures can be found in the appendix.
The results are shown in Tab. 3, which demonstrates that while there are some differences in performance, the generalization capability is consistently achieved across different network architectures.
Specifically, the MLP achieves the best generalization performance, with 45% TOP1 accuracy on unseen subjects and 82% on seen subjects, followed closely by 1D CNN and 3D CNN, which yield comparable results.
The Transformer network exhibits the lowest performance, which may be attributed to the fact that Transformer typically needs larger training datasets to exhibit its superiority, whereas our current dataset for visual brain decoding is relatively small, making CNNs and MLPs more effective in this case.

### 4.4 Generalization vs. Subject Similarity

From the experiments in previous sections, we have seen that the network could exhibit obvious generalization capability when enough subjects are used in training.
During our experiments, interestingly, we notice some performance biases of models trained on different gender groups.
Gender is one of the most commonly observed characteristics of sample similarity, which inspires us that the similarity among subjects might impact the model generalization performance.
To be specific, we train three models using data from 50 male subjects, 50 female subjects, and a mixed group of 25 male and 25 female subjects, respectively.
Then, we evaluate these models on unseen male Subj 1 and female Subj 2, and the results are shown in Tab. 4.
One can see that the model trained on male subjects achieves the best retrieval performance on the unseen male subject (Subj 1) with 36% TOP1 accuracy and 60% TOP3 accuracy, but it performs the worst on the unseen female subject (Subj 2) with 25% TOP1 accuracy and 37% TOP3 accuracy.
In contrast, the model trained on female subjects shows the opposite behaviour, performing better on Subj 2 and worse on Subj 1.
Meanwhile, the model trained on the mixed group always obtain the intermediate result on the unseen subjects.
Such results suggest that the generalization capability is related to the similarity among subjects.

Therefore, we further explore this phenomenon by finding similar and dissimilar subjects to Subj 1 and Subj 2.
Utilizing the method described in Sec. 3.3, we identify 20 most similar subjects and 20 least similar subjects to Subj 1, as well as 20 most similar and least similar subjects to Subj 2.
As shown in Tab. 4, the model trained on the 20 subjects most similar to Subj 1 achieves the best performance on Subj 1, with a TOP1 accuracy of 21% and a TOP3 accuracy of 36%.
In contrast, the model trained on the 20 most dissimilar subjects performs significantly worse, with a TOP1 accuracy of 8% and a TOP3 accuracy of 14%.
A similar trend can be observed for Subj 2, where the model trained on the 20 most similar / dissimilar subjects achieves the highest / lowest performance.
Additionally, the model trained on Subjs 11-30, as a reference for randomly selected 20 subjects, yields moderate performance on both Subj 1 and Subj 2.
This demonstrates that the similarity between subjects can largely affect the generalization performance.
Even with 20 subjects in training, if the subjects are highly dissimilar, the model can achieve little generalization capability, such as 2% TOP1 Acc. on Subj 2.
We also train models on a mixed set of 20 similar and 20 dissimilar subjects for Subj 1 and Subj 2.
The results closely match the performance of models trained on the 20 similar subjects alone.

The above experimental results show that when the subjects are similar, the models achieve better generalization performance, and vice versa.
On the other hand, when a mix of similar and dissimilar subjects are used for training, generalization remains stable, with performance approaching to the models trained on similar subjects.
This suggests that generalization capability depends on learning inherent commonalities among human brains, with substantial tolerance for dissimilarities.
It also explains why increasing the number of subjects enhances generalization — a larger dataset are more likely to include subjects with higher similarities.

**Table 4: Results of models trained on subjects that have different gender or similarity. All the training subjects are from Subj 3-167. Best and worst results are marked by red and blue.**
| Training Subjects | TOP1 Acc. on<br>Subj 1 (male) | TOP3 Acc. on<br>Subj 1 (male) | TOP1 Acc. on<br>Subj 2 (female) | TOP3 Acc. on<br>Subj 2 (female) |
| --- | --- | --- | --- | --- |
| 50 male | 36% | 60% | 25% | 37% |
| 50 female | 27% | 40% | 30% | 42% |
| 25 male + 25 female | 32% | 49% | 27% | 40% |
| 20 similar to Subj 1 | 21% | 36% | - | - |
| 20 dissimilar to Subj 1 | 8% | 14% | - | - |
| \cdashline1-5<br>20 similar to Subj 2 | - | - | 21% | 31% |
| 20 dissimilar to Subj 2 | - | - | 2% | 7% |
| \cdashline1-5<br>Subjs 11-30 (20) | 17% | 31% | 18% | 23% |
| 20 similar + 20 dissimilar to Subj 1 | 21% | 30% | - | - |
| 20 similar + 20 dissimilar to Subj 2 | - | - | 20% | 28% |

### 4.5 Training Strategy

**Table 5: Results of models trained with different training strategies. TOP1 Acc. is averaged over unseen subjects (Subjs 1-10). TOP1 Acc. (seen) is averaged over all seen subjects during training.**
| Trainging Subjects | Subject Number | Strategies | TOP1 Acc. | TOP1 Acc. (seen) |
| --- | --- | --- | --- | --- |
| Subj 11 | 1 | CLIP / BiMixCo+SoftCLIP | 2% / 6% | 79% / 83% |
| Subjs 11-177 | 167 | CLIP / BiMixCo+SoftCLIP | 45% / 50% | 82% / 84% |

In our main experiments, in order to prove that the generalization capability does not come from some specific strategies, we use the simplest contrastive learning pipeline and the CLIP loss to verify our approach on commonly used network architectures.
In this section, we demonstrate that the model performance can be further enhanced if stronger training strategies can be employed. In particular, we adopt the BiMixCo+SoftCLIP training strategy from MindEye1 (Scotti et al., ).
The BiMixCo+SoftCLIP strategy incorporates a data augmentation technique, extending the mixup approach with the InfoNCE loss (He et al., ).
Additionally, we replace the CLIP loss with the SoftCLIP loss, which leverages softmax probability distributions rather than hard labels.
Details of these methods can be found in the appendix.

As shown in Tab. 5, the adoption of BiMixCo+SoftCLIP leads to a noticeable improvement in generalization performance for both single-subject models (TOP1 accuracy improves from 2% to 6%) and multiple-subject models (TOP1 accuracy improves from 45% to 50%).
This strategy also enhances the retrieval accuracy on seen subjects.
These results suggest that better learning strategies can be designed to boost the model generalization capabilities, highlighting the potential of our approach for visual brain decoding.

### 4.6 Experimental Results on the NSD Dataset

**Table 6: Results of models trained on NSD dataset. As in previous works, we randomly extracted 300 pairs from 1,000 pairs of the test set to perform the testing, and run 30 times the experiments to take the average. ‘MindEye1’ means our implemented model of MindEye1 (Scotti et al., ).**
|  |  |  | NSDGeneral Data | NSD Whole-brain Data |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Models | Training Subjects | Testing Subjects | Data Format | TOP1 Acc. | Data Format | TOP1 Acc. |
| MindEye1 | Subj 1 | Subj 1 | 1 × 15,724 | 85% | 83 × 104 × 81 | 35% |
| Ours | Subj 1 | Subj 1 | 1 × 18,000 | 86% | 113 × 136 × 113 | 46% |
| MindEye1 | Subj 7 | Subj 7 | 1 × 12,682 | 70% | 81 × 95 × 78 | 23% |
| Ours | Subj 7 | Subj 7 | 1 × 18,000 | 70% | 113 × 136 × 113 | 29% |
| Ours | Subjs 1,3,4,6,7,8 | Subj 1 | 1 × 18,000 | 83% | 113 × 136 × 113 | 49% |
| Ours | Subjs 1,3,4,6,7,8 | Subj 7 | 1 × 18,000 | 69% | 113 × 136 × 113 | 35% |
| Ours | Subjs 1,3,4,6,7,8 | Subjs 2,5 | 1 × 18,000 | 1% | 113 × 136 × 113 | 1% |

To demonstrate the flexibility of our pipeline, we also train the model on the NSD dataset, including both NSDGeneral and whole-brain data.
The results are shown in Tab. 6. We see that the models trained on Subj 1 using NSDGeneral data with both the original data format (*i.e.*, 1 × 15,724 in MindEye1) and our normalized data format (*i.e.*, 1 × 18,000) achieve similar TOP1 accuracy, *i.e.*, 85% and 86%, respectively.
(The TOP1 accuracy reported in the original paper of MindEye1 (Scotti et al., ) is 84%.)
The results on Subj 7 can yield similar conclusion.
This demonstrates that our pipeline can be well applied to the individual-specific scenario using NSDGeneral data.
However, by using simple interpolation based upsampling to normalize the data format as a 1 × 18,000 vector, we can train models on multiple subjects with different original NSDGeneral data sizes. As shown in the bottom three rows of Tab. 6, by training on subjs 1,3,4,6,7,8 and testing on subj 1 or subj 7, 83% and 69% TOP1 accuracy can still be obtained since subj 1 or subj 7 are included in the training data. However, when testing on the unseen subjs 2 and 5, only 1% TOP1 accuracy is obtained. This is because the NSD dataset has only 8 subjects in total, which is too few to ensure the model generalization performance.

Let’s then evaluate the models trained with whole-brain data. The up right panel of Tab. 6 shows the results on Subj 1 and Subj 7 by MindEye1, which uses the original whole-brain data format, and our model, which uses the normalized data format (*i.e.*, 113 × 136 × 113).
We see that in the case of whole-brain data, the simple normalization of data size can improve much the TOP1 accuracy from 35% to 46% for subj 1 and from 23% to 29% for subj 7. The accuracy is lower than that on NSDGeneral data because the NSDGeneral data are manually labeled brain visual regions.
Again, the normalized data size enables us to train models on multiple subjects with different original data sizes, which cannot be done by MindEye1.
As shown in the bottom right panel of Tab. 6, our model trained on six subjects achieves 49% TOP1 accuracy on Subj 1 and 35% on Subj 7, outperforming single-subject models trained on Subj 1 (46%) and Subj 7 (29%), respectively.
In contrast, on NSDGeneral data, the multiple-subject models perform slightly worse than their single-subject counterparts.
This discrepancy highlights the individual-specific nature of NSDGeneral data, as discussed in Sec. 3.1 and shown in Fig. 3.
We also report the generalization performance of models trained on the six subjects on unseen Subjs 2 and 5.
Again, the model shows weak generalization ability due to the small number of training subjects in NSD dataset.

## 5 Conclusion

Previous visual brain decoding studies typically focused on individual subjects, or training with multiple-subjects but decoding on seen subjects, while little work has been done on exploring the possibility of generalizing visual brain decoding to unseen subjects.
We made an attempt to achieve this goal by leveraging a large dataset from the Human Connectome Project (HCP), constructing 177 × 3,127 image-fMRI pairs from 177 subjects.
Using this dataset, we proposed a learning paradigm, which utilized whole-brain data and a simple and uniform pipeline for processing all subjects, without requiring individual-specific adaptations.
Via extensive experiments, we found that the model generalization capability emerged with the increase of training subjects, and such generalization capability held across different network architectures. In addition, the similarity between subjects also played a role in improving the generalization capability.
These findings revealed the inherent similarity in brain activities across individuals, which has significant implications for future studies.
As larger, more diverse datasets may become available, this work can provide basis for training a brain encoding foundation model in the future.

## A The Details of Architectures

In the main paper, we validated the generalization capability of visual brain decoding models on unseen subjects across popular network architectures, including MLP, CNN and Transformer.
We show the detailed network structures in this section.
The input to the networks is fMRI voxels of size 113 × 136 × 113.

**Table A.1: Detailed architecture of our MLP network.**
| Layer Type | Input Size | Output Size | Kernel/Linear Size | Stride/Padding |
| --- | --- | --- | --- | --- |
| Conv3D (conv1) | $1\times 113\times 136\times 113$ | $32\times 38\times 46\times 38$ | $9\times 9\times 9$ | Stride 3, Padding 4 |
| BatchNorm3D (bn1) | $32\times 38\times 46\times 38$ | $32\times 38\times 46\times 38$ | - | - |
| Conv3D (conv2) | $32\times 38\times 46\times 38$ | $48\times 19\times 23\times 19$ | $7\times 7\times 7$ | Stride 2, Padding 3 |
| BatchNorm3D (bn2) | $48\times 19\times 23\times 19$ | $48\times 19\times 23\times 19$ | - | - |
| Conv3D (conv3) | $48\times 19\times 23\times 19$ | $64\times 10\times 12\times 10$ | $5\times 5\times 5$ | Stride 2, Padding 2 |
| BatchNorm3D (bn3) | $64\times 10\times 12\times 10$ | $64\times 10\times 12\times 10$ | - | - |
| ReLU (activation) | $64\times 10\times 12\times 10$ | $64\times 10\times 12\times 10$ | - | - |
| Flatten | $64\times 10\times 12\times 10$ | $76800$ | - | - |
| Linear (lin0) | $76800$ | $4096$ | - | - |
| MLP (4 layers) | $4096$ | $4096$ | - | - |
| Linear (lin1) | $4096$ | $257\times 1024$ | - | - |

#### MLP.

Tab. A.1 presents the architecture of our MLP network.
We first employ a 3-layer 3D CNN for feature extraction, followed by a fully connected layer to reduces the feature to a 1D vector of 4096 elements.
Similar to MindEye1 (Scotti et al., ), we apply the MLP backbone in this feature space with the vector length of 4096.
Finally, we utilize a fully connected layer to expand the feature from 4096 to 257 × 1024, which is used for loss calculation.

**Table A.2: Detailed architecture of our 1D CNN network.**
| Layer Type | Input Size | Output Size | Kernel/Linear Size | Stride/Padding |
| --- | --- | --- | --- | --- |
| Conv3D (conv1) | $1\times 113\times 136\times 113$ | $32\times 38\times 46\times 38$ | $9\times 9\times 9$ | Stride 3, Padding 4 |
| BatchNorm3D (bn1) | $32\times 38\times 46\times 38$ | $32\times 38\times 46\times 38$ | - | - |
| Conv3D (conv2) | $32\times 38\times 46\times 38$ | $48\times 19\times 23\times 19$ | $7\times 7\times 7$ | Stride 2, Padding 3 |
| BatchNorm3D (bn2) | $48\times 19\times 23\times 19$ | $48\times 19\times 23\times 19$ | - | - |
| Conv3D (conv3) | $48\times 19\times 23\times 19$ | $64\times 10\times 12\times 10$ | $5\times 5\times 5$ | Stride 2, Padding 2 |
| BatchNorm3D (bn3) | $64\times 10\times 12\times 10$ | $64\times 10\times 12\times 10$ | - | - |
| ReLU (activation) | $64\times 10\times 12\times 10$ | $64\times 10\times 12\times 10$ | - | - |
| Flatten | $64\times 10\times 12\times 10$ | $76800$ | - | - |
| Linear (lin0) | $76800$ | $4096$ | - | - |
| Reshape | $4096$ | $1\times 4096$ | - | - |
| Conv1D (conv01) | $1\times 4096$ | $64\times 680$ | Kernel 13 | Stride 6 |
| BatchNorm1D (bn01) | $64\times 680$ | $64\times 680$ | - | - |
| Conv1D (conv02) | $64\times 680$ | $128\times 134$ | Kernel 11 | Stride 5 |
| BatchNorm1D (bn02) | $128\times 134$ | $128\times 134$ | - | - |
| Conv1D (conv03) | $128\times 134$ | $256\times 32$ | Kernel 9 | Stride 4 |
| BatchNorm1D (bn03) | $256\times 32$ | $256\times 32$ | - | - |
| Conv1D (conv04) | $256\times 32$ | $512\times 9$ | Kernel 7 | Stride 3 |
| BatchNorm1D (bn04) | $512\times 9$ | $512\times 9$ | - | - |
| ResBlock1D (x8) | $512\times 9$ | $512\times 9$ | - | - |
| Flatten | $512\times 9$ | $4608$ | - | - |
| Linear (lin1) | $4608$ | $257\times 1024$ | - | - |

#### 1D CNN.

As shown in Tab. A.2, similar to our MLP network, our 1D CNN network first reduces the feature to a vector of length 4096.
Then we use several separate 1D CNN layers and eight 1D ResBlock layers for feature mapping, which constitute our 1D CNN backbone.
Finally, a fully connected layer is used to produce the output with size of 257 × 1024.

**Table A.3: Detailed architecture of our 3D CNN network.**
| Layer Type | Input Size | Output Size | Kernel/Linear Size | Stride/Padding |
| --- | --- | --- | --- | --- |
| Conv3D (conv1) | $1\times 113\times 136\times 113$ | $32\times 38\times 46\times 38$ | $9\times 9\times 9$ | Stride 3, Padding 4 |
| BatchNorm3D (bn1) | $32\times 38\times 46\times 38$ | $32\times 38\times 46\times 38$ | - | - |
| Conv3D (conv2) | $32\times 38\times 46\times 38$ | $48\times 19\times 23\times 19$ | $7\times 7\times 7$ | Stride 2, Padding 3 |
| BatchNorm3D (bn2) | $48\times 19\times 23\times 19$ | $48\times 19\times 23\times 19$ | - | - |
| Conv3D (conv3) | $48\times 19\times 23\times 19$ | $64\times 10\times 12\times 10$ | $5\times 5\times 5$ | Stride 2, Padding 2 |
| BatchNorm3D (bn3) | $64\times 10\times 12\times 10$ | $64\times 10\times 12\times 10$ | - | - |
| Conv3D (conv4) | $64\times 10\times 12\times 10$ | $90\times 5\times 6\times 5$ | $5\times 5\times 5$ | Stride 2, Padding 2 |
| BatchNorm3D (bn4) | $90\times 5\times 6\times 5$ | $90\times 5\times 6\times 5$ | - | - |
| Conv3D (conv5) | $90\times 5\times 6\times 5$ | $150\times 3\times 3\times 3$ | $5\times 5\times 5$ | Stride 2, Padding 2 |
| BatchNorm3D (bn5) | $150\times 3\times 3\times 3$ | $150\times 3\times 3\times 3$ | - | - |
| ReLU (activation) | $150\times 3\times 3\times 3$ | $150\times 3\times 3\times 3$ | - | - |
| ResBlock3D (x8) | $150\times 3\times 3\times 3$ | $150\times 3\times 3\times 3$ | - | - |
| Flatten | $150\times 3\times 3\times 3$ | $4050$ | - | - |
| Linear (lin1) | $4050$ | $257\times 1024$ | - | - |

#### 3D CNN.

Different from the above MLP and 1D CNN networks, as shown in Tab. A.3, our 3D CNN network uses 3D CNN layers and 3D ResBlock layers to process the input.
After the main processing, a fully connected layer is used to produce the final output with size of 257 × 1024.

**Table A.4: Detailed architecture of our Transformer network.**
| Layer Type | Input Size | Output Size | Kernel/Linear Size | Stride/Padding |
| --- | --- | --- | --- | --- |
| Conv3D (conv1) | $1\times 113\times 136\times 113$ | $32\times 38\times 46\times 38$ | $9\times 9\times 9$ | Stride 3, Padding 4 |
| BatchNorm3D (bn1) | $32\times 38\times 46\times 38$ | $32\times 38\times 46\times 38$ | - | - |
| Conv3D (conv2) | $32\times 38\times 46\times 38$ | $48\times 19\times 23\times 19$ | $7\times 7\times 7$ | Stride 2, Padding 3 |
| BatchNorm3D (bn2) | $48\times 19\times 23\times 19$ | $48\times 19\times 23\times 19$ | - | - |
| Conv3D (conv3) | $48\times 19\times 23\times 19$ | $64\times 10\times 12\times 10$ | $5\times 5\times 5$ | Stride 2, Padding 2 |
| BatchNorm3D (bn3) | $64\times 10\times 12\times 10$ | $64\times 10\times 12\times 10$ | - | - |
| ReLU (activation) | $64\times 10\times 12\times 10$ | $64\times 10\times 12\times 10$ | - | - |
| Flatten | $64\times 10\times 12\times 10$ | $76800$ | - | - |
| Linear (lin0) | $76800$ | $4096$ | - | - |
| Reshape | $4096$ | $16\times 256$ | - | - |
| Transformer (d=256, h=8) (x24) | $16\times 256$ | $16\times 256$ | - | - |
| Reshape | $16\times 256$ | $4096$ | - | - |
| Linear (lin1) | $4096$ | $257\times 1024$ | - | - |

#### Transformer.

The details of our Transformer network are shown in Tab. A.4. Similar to our MLP and 1D CNN networks, our Transformer network first reduces the feature to a 4096 sized vector.
Then we reshape the feature to 16 × 256, which is then sent to the Transformer backbone with 24 Transformer block layers.
After that, we reshape the output of Transformer backbone back to 4096 and employ a fully connected layer to produce the final output with size of 257 × 1024.

Figure: Figure A.1: The visualized image retrieval results. ‘Target images’ refer to those viewed by Subj 1 from the HCP dataset, while ‘Retrieved images’ represent the corresponding retrieval outputs from the visual brain decoding model trained on 167 subjects. The retrieved images are ranked from left to right based on retrieval similarity.
Refer to caption: x5.png

## B The Details of Training Strategy

In our main paper, we adopt the BiMixCo+SoftCLIP training strategy from MindEye1 (Scotti et al., ) to further enhance our visual brain decoding network.
In this section, we provide the details of this strategy, which uses BiMixCo for one-third of the training and employs SoftCLIP for the rest of training.

#### BiMixCo.

The BiMixCo combines MixCo (Kim et al., ) (an extension of mixup that uses the InfoNCE loss) and the bidirectional CLIP loss.
To be specific, as described in MindEye1 (Scotti et al., ), voxels are mixed by a factor $\lambda$ sampled from the Beta distribution with $\alpha=\beta=0.15$:

$$ $\displaystyle x_{\text{mix}_{i,k_{i}}}=\lambda_{i}\cdot x_{i}+(1-\lambda_{i}) \cdot x_{k_{i}},\quad p_{i}^{*}=f(x_{\text{mix}_{i,k_{i}}}),\quad p_{i}=f(x_{i }),\quad t_{i}=\text{CLIP}_{\text{Image}}(y_{i}),$ (4) $$

where $x_{i}$ and $y_{i}$ represent the $i$-th fMRI voxel and image, respectively. $k_{i}\in[1,N]$ is an arbitrary mixing index for the $i$-th datapoint and $f$ represents the decoding network. $p^{*}$, $p$ and $t$ are $L_{2}$-normalized. The CLIP loss with MixCo is defined as:

$$ $\displaystyle\mathcal{L}_{\text{BiMixCo}}=-\sum_{i=1}^{N}\left[\lambda_{i} \cdot\log\left(\frac{\exp\left(\frac{p_{i}^{*}\cdot t_{i}}{\tau}\right)}{\sum_ {m=1}^{N}\exp\left(\frac{p_{i}^{*}\cdot t_{m}}{\tau}\right)}\right)+(1-\lambda _{i})\cdot\log\left(\frac{\exp\left(\frac{p_{i}^{*}\cdot t_{k_{i}}}{\tau} \right)}{\sum_{m=1}^{N}\exp\left(\frac{p_{i}^{*}\cdot t_{m}}{\tau}\right)} \right)\right]$ $\displaystyle-\sum_{j=1}^{N}\left[\lambda_{j}\cdot\log\left(\frac{\exp\left( \frac{p_{j}^{*}\cdot t_{j}}{\tau}\right)}{\sum_{m=1}^{N}\exp\left(\frac{p_{m}^ {*}\cdot t_{j}}{\tau}\right)}\right)+\sum_{\{l\mid k_{l}=j\}}(1-\lambda_{l}) \cdot\log\left(\frac{\exp\left(\frac{p_{l}^{*}\cdot t_{j}}{\tau}\right)}{\sum_ {m=1}^{N}\exp\left(\frac{p_{m}^{*}\cdot t_{j}}{\tau}\right)}\right)\right],$ (5) $$

where $\tau$ is a temperature hyperparameter, and $N$ is the batch size.

#### SoftCLIP.

The soft contrastive loss (Scotti et al., ) takes the dot product of CLIP image embeddings within a batch to generate the soft labels.
The loss (we omit the bidirectional component for brevity) is calculated between CLIP-CLIP and Brain-CLIP matrices as:

$$ $\displaystyle\mathcal{L}_{\text{SoftCLIP}}=-\sum_{i=1}^{N}\sum_{j=1}^{N}\left[ \frac{\exp\left(\frac{t_{i}\cdot t_{j}}{\tau}\right)}{\sum_{m=1}^{N}\exp\left( \frac{t_{i}\cdot t_{m}}{\tau}\right)}\cdot\log\left(\frac{\exp\left(\frac{p_{i }\cdot t_{j}}{\tau}\right)}{\sum_{m=1}^{N}\exp\left(\frac{p_{i}\cdot t_{m}}{ \tau}\right)}\right)\right].$ (6) $$

## C The Visualized Image Retrieval Results

Fig. A.1 depicts the image retrieval results of our visual brain decoding model, which is trained on 167 subjects from our consolidated HCP dataset.
The results are extracted from the testing of Subject 1, which is an unseen subject to this model.
We show the top three images that have the highest retrieval similarities to the target image seen by Subject 1.
We can see from Fig. A.1 that the retrieved images exhibit significant visual similarity to the target image.
For example, in the first row, the target and retrieved images are all facial images, while the first retrieved image is exactly the target image. In the fourth row, all the retrieved images have two persons, as in the target image. In the last two rows, the retrieved images share similar colors and tones to the target images.
Nonetheless, in some cases, such as the second retrieved image in the fifth row and the third retrieved image in the sixth row, the retrieval results are not satisfactory.

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: I don't really agree with your statement in lines 506-508 of Section 4.6 that the poor generalization of the method on the NSD dataset is due to the problem of too few subjects. It seems to me at the moment that the integrated new dataset is consistent across all visual stimuli viewed by all subjects. So even for unseen subjects, the model is trained with the images that unseen subjects 'see'. However, for the NSD dataset, the images seen by different subjects, although they are all COCO images, are not the same in the training set. Therefore, in the absence of any comparison of related methods, how can it be proved that the method proposed in this paper is better not due to the difference in dataset ?
soundness: 3
strengths: 1. This paper is well written and easy for the reader to read.
2. The motivation for the zero-shot setting is interesting.
3. The experimental design is very rich with multiple validations. Readers are inspired by explorations of the impact of gender.
4. In a sense, integrating the new dataset breaks the routine of NSD data use and is contributory.
summary: This paper presents a new brain decoding paradigm for zero-shot brain decoding. Multiple experimental design exposes that brain decoding models can be generalized from existing subjects to unseen subjects.The effect of inter-subject similarity, such as gender, on generalization ability is discussed for the first time.
weaknesses: 1. From the point of view of machine learning, the algorithms for brain decoding in this paper are basically existing methods. The use of downsampling to obtain voxels of uniform size across multiple subjects has been used, e.g. CLIP-MUSED [1] uses PCA downsampling to align different subjects, also without additional training. 
2. The methodology mentioned by the authors was not compared to any relevant studies quantitatively. The methods mentioned by the authors were not compared to any relevant studies, so it is difficult to comparatively measure whether the methods proposed by the authors are more effective or not. There are actually a subset of papers that have performed the appropriate anatomical alignment for zero-shot brain decoding[2, 3, 4]. Because of open source issues, a full reproduction may be a bit difficult, but small designs such as the use of **fsaverage** are easy to implement (fsaverage is already widely used in neuroscience). 
3. While the new integrated dataset makes sense, it seems to me that a neuroscience research task without algorithmic design requires a dataset of its own design. I think this is missing in this paper. And the decoding task is very simple, containing only the retrieval task. Although the authors mention that the method can be generalized to the reconstruction task, a glance at related papers on the same method makes it hard for me to be convinced (the same method using downsampling fails miserably on the reconstruction task).

Overall I do not deny the contribution of this paper and I think the research ideas in this paper are very interesting and deserve to be published. The reason I am currently leaning towards rejection is that I am concerned that the paper may be more suited to brain science related journals or conferences than machine learning conferences like **ICLR**. Because the methods in this paper do not involve any representation learning methods or any algorithmic innovations, and the conclusions are more of a scientific experimental nature. 

I would consider changing the rating to a positive one if the authors could convince me very well.

**Reference**
1. Zhou, Qiongyi, et al. "CLIP-MUSED: CLIP-Guided Multi-Subject Visual Neural Information Semantic Decoding." ICLR 2024.
2. Thual, Alexis, et al. "Aligning brain functions boosts the decoding of visual semantics in novel subjects." ArXiv 2023.
3. Bao, Guangyin, et al. "Wills Aligner: A Robust Multi-Subject Brain Representation Learner." Arxiv 2024.
4. Qian, Xuelin, et al. "fmri-pte: A large-scale fmri pretrained transformer encoder for multi-subject brain activity decoding." Arxiv 2023.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 3
questions: Please provide a detailed rationale explaining why adding more subjects is beneficial. Specifically, how does this approach address the alignment problem? A clear explanation would help clarify the method's impact on the cross-subject variability and functional alignment issues inherent to neuroAI.

Please compare the performance of the proposed method with at least two other state-of-the-art approaches. This comparison would help establish its relative effectiveness and efficiency, particularly in terms of computational costs, accuracy, and robustness across subjects.
soundness: 3
strengths: The writing is clear, and the rationale is logical. The question of alignment across subjects, which the paper addresses, is timely and aligns with topics discussed at prominent venues.
summary: The paper explores cross-subject visual brain decoding, specifically focusing on scenarios where no data from unseen subjects are included in the training model. The contributions are twofold: first, the authors propose an effective, straightforward approach to generate image-fMRI data at scale using the HCP video viewing dataset. Second, they introduce a decoding method based on CLIP encoding combined with cosine similarity decoding. Although the approach is admittedly simple, it is practical and well-justified.
weaknesses: The decoding methodology is likely the paper’s most relevant contribution to ICLR, though I have identified two significant limitations in this area.

Firstly, the authors show that incorporating data from additional subjects in training improves the model's performance, interpreting this as a solution to the alignment problem in neuroAI. In this field, we frequently encounter issues with requiring identical input-output pairs across subjects. While the authors suggest that their method mitigates this problem, they do not explain how. Is it possible that the model simply learns shared noise inherent to the videos across subjects as the data pool increases? We know that functional alignment remains a challenge because brain responses vary at the voxel level, even if the encoded information is similar across the whole region. It is unclear how, if at all, the proposed method addresses this challenge.

Secondly, the paper lacks a direct comparison with other state-of-the-art methods, which the authors partially review in the introduction. Could alternative methods achieve similar or even better performance? The authors suggest that other methods are computationally expensive in terms of parameters and training, yet they do not provide specific details. Quantitative comparisons and a more in-depth analysis of parameter costs are essential to substantiate this claim.

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: 1. The use of a video dataset to define an image retrieval task is interesting, however the distribution of images obtained by sampling frames every 1 s might be biased in a way that influences the interpretation of the results. Given frames within a scene are likely significantly correlated, the level of difficulty of the retrieval task on the test set will vary a lot if examples are strictly sampled from different scenes or not. Were the test images sampled uniformly across the 3127 images? A helpful visualization might be a grid of all sampled test images.
2. Similarly, if a frame $i_j$ from was selected to be part of the test set and the surrounding frames that are part of the same scene (e.g. $i_{j-1}$ and $i_{j+1}$, which could be very similar to $i_j$ if the visual stimulus changes slowly) end up in the training set, there is a strong potential for overfitting. Could this be the case here?
3. Averaging the 4-s window after the target frame is likely suboptimal as this misses the second half of the peak response (occuring at 4-s according to the reference cited). Have the authors looked at different pooling strategies (e.g. averaging seconds 2 to 6)?
4. How many parameters do the different model configurations have? How were the hyperparameters selected? For instance I find it surprising that such a large Transformer was used (24 layers according to Suppl. Figure 4).
5. What upsampling methodology was used?
6. Previous studies on the NSD dataset relied on precomputed beta values obtained through a GLM fitting procedure that help improve SNR. Can the authors confirm whether the “whole-brain” data they used is also betas or is it BOLD signals? This could also be a more likely explanation for the discrepancy between results on NSDGeneral and “Whole-brain” data in Table 6.
7. How were the images of Suppl Figure 1 selected? Are these retrieval results representative of the other images in the test set?
8. Other comments:
- At line 39, Benchetrit et al. 2023 is cited as an example of fMRI visual decoding, however they used MEG.
- In the last sentence of the conclusion, the authors probably meant to say “a brain decoding foundation model” (instead of “encoding”) as brain encoding would instead mean reconstructing brain activity from the stimuli.
soundness: 2
strengths: * Originality: While the proposed methodology is similar to previous work, the paper studies a setting (generalization across multiple subjects) which has not been explored much before. The analysis of the impact of training data composition (gender bias and of train-test set similarity) is also interesting and original.
* Quality: The paper is of good quality and presents relevant experiments to support the claims made by the authors.
* Clarity: The paper is overall well and clearly written, and the problem setting is well introduced and supported.
* Significance: The results are promising and motivate further scaling up of brain activity datasets.
summary: The paper explores the generalization properties of visual decoding models when trained and tested on different subjects. Different model architectures (3D ConvNet, 1D ConvNet, Transformer, MLP) were trained on subsets of a dataset containing fMRI brain activity data collected while subjects watched audio-visual movies. Models were trained to recover the seen video frame out of a batch of images. The use of an upsampling step made it possible to reuse a single model across all subjects (vs. subject-specific layer in previous work). Results show that model performance increases as more subjects are seen during training, and that distribution shift between training and test subjects can negatively affect performance.
weaknesses: 1. The description of the dataset curation and test set sampling procedure lacks details (see Q1-2). As movie frames are likely correlated and do not cover the same semantic space as “standard” brain-image datasets (in which the images are selected to cover varied categories, e.g. NSD), it is unclear whether the models might in fact overfit to the specific scenes seen in the training set. This could have an impact on the absolute performance values reported in the paper.
2. There is also missing information about the architectures (upsampling procedure, number of parameters, hyperparameter selection procedure; see Q4-5).

---

confidence: 4
contribution: 1
initial_rating: 1
presentation: 2
questions: In addition to addressing the previously mentioned weaknesses, the following questions should be clarified to strengthen the paper:

Are there any validation checks that confirm whether the fMRI signals accurately reflect the intended neural responses to individual images?

Are there specific characteristics of whole-brain data that introduce redundancy or noise, which might explain the superior results of the hand-picked voxel approach?

In lines 66-67, it is stated that “the model becomes increasingly complex as the number of subjects increases, making it challenging to scale to a larger population,” referring to state-of-the-art architectures. It would be valuable to include a comparison of model size (in terms of the number of parameters) between the proposed approach and existing architectures. Such a comparison would help highlight the scalability advantages of the proposed model more clearly.
soundness: 1
strengths: The paper aims to address two major challenges in the field of visual brain decoding: the scarcity of existing datasets and the potential scalability issues of current pipelines when expanding to a larger number of subjects. However, the evidence provided for the proposed method is insufficient to fully support its effectiveness in tackling these problems.
summary: The paper aims to tackle the challenge of visual brain decoding with a focus on improving generalization across a broader population of subjects. The authors generate a new dataset based on the HCP dataset, where image-fMRI pairs are derived from the last frame of each second in a movie, with the fMRI signals averaged over a specific time window. To map brain signals to the CLIP embeddings of corresponding images, they employ a contrastive learning approach. The discrepancy in voxel counts among subjects is addressed through a straightforward upsampling technique. Furthermore, the authors propose a novel architecture designed to mitigate the exponentially increasing model size typically required by state-of-the-art methods when integrating additional subjects into the pipeline.
weaknesses: 1) I have concerns regarding the dataset generation methodology. It appears that the fMRIs generated for each image may not accurately capture the brain's response to the specific features of those images. Specifically, the resulting fMRI signals seem to reflect an accumulation of activity from prior stimuli, rather than providing a clear, one-to-one representation of the neural response to individual image features. This raises questions about whether the fMRI data truly aligns with the visual stimuli as intended.

2) The validity of the generated dataset is questionable, particularly given the notable performance disparity observed between the NSD dataset and the paper's dataset (as noted in Table 2, row 1, compared to Table 6, row 2). It is unclear why there is such a significant difference, raising concerns about potential biases or inconsistencies in the data generation process that may not accurately reflect the neural representation of the images.

3) In line 485, the paper reports MindEye 1's top-1 accuracy as 84%. However, according to Scotti et al. (2024a), MindEye 1 achieved a top-1 accuracy of 93.1% for image retrieval and 90.1% for brain retrieval (Table 1, row 5). In contrast, Scotti et al. (2024b) present MindEye 1's results as an average across four individual models (Table 1, row 3). Therefore, despite the mention of "our implementation" in line 488 which is unclear, it appears that this approach does not surpass the current state-of-the-art performance.

4) Lines 51-52 suggest that hand-picking voxels is not ideal for data collection, yet the results in Table 6 (columns 5 and 7) demonstrate the superiority of this approach. This discrepancy raises questions about the claim. It is plausible that 'whole-brain' data could contain a significant amount of redundancy or noise, as suggested by prior research (Lin et al. "Redundancy and dependency in brain activities." Shared Visual Representations in Human & Machine Intelligence (2022).), which may explain the better performance of the hand-picked voxel strategy.

5) Table 6, row 7, indicates extremely poor performance, raising doubts about the pipeline’s ability to generalize effectively. In contrast, Scotti et al. (2024b), Table 1, row 7, report significantly better results even with a subset of the data. To strengthen the evaluation, a clear comparison between the proposed approach and MindEye2 is essential, as it would help clarify the relative effectiveness of the proposed method.

6) As the authors noted, previous studies have employed various methods to address the discrepancies in voxel counts across subjects, such as creating a common representational space (e.g., Scotti et al., 2024b). However, it is unclear how a simple upsampling approach effectively manages the differences in fMRI signals among subjects. More details are needed on the implementation of the upsampling process. Additionally, an in-depth analysis comparing upsampling to existing alignment methods (e.g., functional alignment, anatomical alignment, common representational space) is necessary to understand whether it can truly serve as a viable alternative.

Meta-review: This submission contributes a method for brain-based decoding generalizing across subjects. The study is a vision task, mapping brain signals to CLIP embeddings of the images. The submission generated interest and discussion from the reviewers, in particular the across-subject generalization. The reviewers appreciated the empirical design, however the comparison to baselines is not clear at all, which is very unfortunate. The other should strive to address this last point, clarifying the corresponding points raised in the discussion and ideally running experiments to complement them.

Decision: Accept (Poster)