Toward Generalizing Visual Brain Decoding to Unseen Subjects

Authors: Xiangtao Kong, Kexin Huang, Ping Li, Lei Zhang
Primary area: applications to neuroscience & cognitive science
Keywords: Visual brain decoding, fMRI - image retrieval, Generalizing to unseen subjects

Abstract:
Visual brain decoding aims to decode visual information from human brain activities. Despite the great progress, one critical limitation of current brain decoding research lies in the lack of generalization capability to unseen subjects. Prior work typically focuses on decoding brain activity of individuals based on the observation that different subjects exhibit different brain activities, while it remains unclear whether brain decoding can be generalized to unseen subjects. This study aims to answer this question. We first consolidate an image-fMRI dataset consisting of stimulus-image and fMRI-response pairs, involving 177 subjects in the movie-viewing task of the Human Connectome Project (HCP). This dataset allows us to investigate the brain decoding performance with the increase of participants. We then present a learning paradigm that applies uniform processing across all subjects, instead of employing different network heads or tokenizers for individuals as in previous methods, so that we can accommodate a large number of subjects to explore the generalization capability across different subjects. A series of experiments are conducted and we have the following findings. First, the network exhibits clear generalization capabilities with the increase of training subjects. Second, the generalization capability is common to popular network architectures (MLP, CNN and Transformer). Third, the generalization performance is affected by the similarity between subjects. Our findings reveal the inherent similarities in brain activities across individuals. With the emergence of larger and more comprehensive datasets, it is possible to train a brain decoding foundation model in the future. Codes and models can be found at https://github.com/Xiangtaokong/TGBD}{https://github.com/Xiangtaokong/TGBD.

Decision: accept (poster)
Reviewer scores: 6, 6, 6, 5

Reviews and discussion:
<thread>
creation date:
1730309116546

thread_type:
official_review

<thread_object>
creator: reviewer_cdxj

<content>
summary:
The paper aims to tackle the challenge of visual brain decoding with a focus on improving generalization across a broader population of subjects. The authors generate a new dataset based on the HCP dataset, where image-fMRI pairs are derived from the last frame of each second in a movie, with the fMRI signals averaged over a specific time window. To map brain signals to the CLIP embeddings of corresponding images, they employ a contrastive learning approach. The discrepancy in voxel counts among subjects is addressed through a straightforward upsampling technique. Furthermore, the authors propose a novel architecture designed to mitigate the exponentially increasing model size typically required by state-of-the-art methods when integrating additional subjects into the pipeline.

strengths:
The paper aims to address two major challenges in the field of visual brain decoding: the scarcity of existing datasets and the potential scalability issues of current pipelines when expanding to a larger number of subjects. However, the evidence provided for the proposed method is insufficient to fully support its effectiveness in tackling these problems.

weaknesses:
1) I have concerns regarding the dataset generation methodology. It appears that the fMRIs generated for each image may not accurately capture the brain's response to the specific features of those images. Specifically, the resulting fMRI signals seem to reflect an accumulation of activity from prior stimuli, rather than providing a clear, one-to-one representation of the neural response to individual image features. This raises questions about whether the fMRI data truly aligns with the visual stimuli as intended.

2) The validity of the generated dataset is questionable, particularly given the notable performance disparity observed between the NSD dataset and the paper's dataset (as noted in Table 2, row 1, compared to Table 6, row 2). It is unclear why there is such a significant difference, raising concerns about potential biases or inconsistencies in the data generation process that may not accurately reflect the neural representation of the images.

3) In line 485, the paper reports MindEye 1's top-1 accuracy as 84%. However, according to Scotti et al. (2024a), MindEye 1 achieved a top-1 accuracy of 93.1% for image retrieval and 90.1% for brain retrieval (Table 1, row 5). In contrast, Scotti et al. (2024b) present MindEye 1's results as an average across four individual models (Table 1, row 3). Therefore, despite the mention of "our implementation" in line 488 which is unclear, it appears that this approach does not surpass the current state-of-the-art performance.

4) Lines 51-52 suggest that hand-picking voxels is not ideal for data collection, yet the results in Table 6 (columns 5 and 7) demonstrate the superiority of this approach. This discrepancy raises questions about the claim. It is plausible that 'whole-brain' data could contain a significant amount of redundancy or noise, as suggested by prior research (Lin et al. "Redundancy and dependency in brain activities." Shared Visual Representations in Human & Machine Intelligence (2022).), which may explain the better performance of the hand-picked voxel strategy.

5) Table 6, row 7, indicates extremely poor performance, raising doubts about the pipeline’s ability to generalize effectively. In contrast, Scotti et al. (2024b), Table 1, row 7, report significantly better results even with a subset of the data. To strengthen the evaluation, a clear comparison between the proposed approach and MindEye2 is essential, as it would help clarify the relative effectiveness of the proposed method.

6) As the authors noted, previous studies have employed various methods to address the discrepancies in voxel counts across subjects, such as creating a common representational space (e.g., Scotti et al., 2024b). However, it is unclear how a simple upsampling approach effectively manages the differences in fMRI signals among subjects. More details are needed on the implementation of the upsampling process. Additionally, an in-depth analysis comparing upsampling to existing alignment methods (e.g., functional alignment, anatomical alignment, common representational space) is necessary to understand whether it can truly serve as a viable alternative.

questions:
In addition to addressing the previously mentioned weaknesses, the following questions should be clarified to strengthen the paper:

Are there any validation checks that confirm whether the fMRI signals accurately reflect the intended neural responses to individual images?

Are there specific characteristics of whole-brain data that introduce redundancy or noise, which might explain the superior results of the hand-picked voxel approach?

In lines 66-67, it is stated that “the model becomes increasingly complex as the number of subjects increases, making it challenging to scale to a larger population,” referring to state-of-the-art architectures. It would be valuable to include a comparison of model size (in terms of the number of parameters) between the proposed approach and existing architectures. Such a comparison would help highlight the scalability advantages of the proposed model more clearly.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730475189493

thread_type:
official_review

<thread_object>
creator: reviewer_qqas

<content>
summary:
The paper explores the generalization properties of visual decoding models when trained and tested on different subjects. Different model architectures (3D ConvNet, 1D ConvNet, Transformer, MLP) were trained on subsets of a dataset containing fMRI brain activity data collected while subjects watched audio-visual movies. Models were trained to recover the seen video frame out of a batch of images. The use of an upsampling step made it possible to reuse a single model across all subjects (vs. subject-specific layer in previous work). Results show that model performance increases as more subjects are seen during training, and that distribution shift between training and test subjects can negatively affect performance.

strengths:
* Originality: While the proposed methodology is similar to previous work, the paper studies a setting (generalization across multiple subjects) which has not been explored much before. The analysis of the impact of training data composition (gender bias and of train-test set similarity) is also interesting and original.
* Quality: The paper is of good quality and presents relevant experiments to support the claims made by the authors.
* Clarity: The paper is overall well and clearly written, and the problem setting is well introduced and supported.
* Significance: The results are promising and motivate further scaling up of brain activity datasets.

weaknesses:
1. The description of the dataset curation and test set sampling procedure lacks details (see Q1-2). As movie frames are likely correlated and do not cover the same semantic space as “standard” brain-image datasets (in which the images are selected to cover varied categories, e.g. NSD), it is unclear whether the models might in fact overfit to the specific scenes seen in the training set. This could have an impact on the absolute performance values reported in the paper.
2. There is also missing information about the architectures (upsampling procedure, number of parameters, hyperparameter selection procedure; see Q4-5).

questions:
1. The use of a video dataset to define an image retrieval task is interesting, however the distribution of images obtained by sampling frames every 1 s might be biased in a way that influences the interpretation of the results. Given frames within a scene are likely significantly correlated, the level of difficulty of the retrieval task on the test set will vary a lot if examples are strictly sampled from different scenes or not. Were the test images sampled uniformly across the 3127 images? A helpful visualization might be a grid of all sampled test images.
2. Similarly, if a frame $i_j$ from was selected to be part of the test set and the surrounding frames that are part of the same scene (e.g. $i_{j-1}$ and $i_{j+1}$, which could be very similar to $i_j$ if the visual stimulus changes slowly) end up in the training set, there is a strong potential for overfitting. Could this be the case here?
3. Averaging the 4-s window after the target frame is likely suboptimal as this misses the second half of the peak response (occuring at 4-s according to the reference cited). Have the authors looked at different pooling strategies (e.g. averaging seconds 2 to 6)?
4. How many parameters do the different model configurations have? How were the hyperparameters selected? For instance I find it surprising that such a large Transformer was used (24 layers according to Suppl. Figure 4).
5. What upsampling methodology was used?
6. Previous studies on the NSD dataset relied on precomputed beta values obtained through a GLM fitting procedure that help improve SNR. Can the authors confirm whether the “whole-brain” data they used is also betas or is it BOLD signals? This could also be a more likely explanation for the discrepancy between results on NSDGeneral and “Whole-brain” data in Table 6.
7. How were the images of Suppl Figure 1 selected? Are these retrieval results representative of the other images in the test set?
8. Other comments:
- At line 39, Benchetrit et al. 2023 is cited as an example of fMRI visual decoding, however they used MEG.
- In the last sentence of the conclusion, the authors probably meant to say “a brain decoding foundation model” (instead of “encoding”) as brain encoding would instead mean reconstructing brain activity from the stimuli.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730650206227

thread_type:
official_review

<thread_object>
creator: reviewer_x87x

<content>
summary:
The paper explores cross-subject visual brain decoding, specifically focusing on scenarios where no data from unseen subjects are included in the training model. The contributions are twofold: first, the authors propose an effective, straightforward approach to generate image-fMRI data at scale using the HCP video viewing dataset. Second, they introduce a decoding method based on CLIP encoding combined with cosine similarity decoding. Although the approach is admittedly simple, it is practical and well-justified.

strengths:
The writing is clear, and the rationale is logical. The question of alignment across subjects, which the paper addresses, is timely and aligns with topics discussed at prominent venues.

weaknesses:
The decoding methodology is likely the paper’s most relevant contribution to ICLR, though I have identified two significant limitations in this area.

Firstly, the authors show that incorporating data from additional subjects in training improves the model's performance, interpreting this as a solution to the alignment problem in neuroAI. In this field, we frequently encounter issues with requiring identical input-output pairs across subjects. While the authors suggest that their method mitigates this problem, they do not explain how. Is it possible that the model simply learns shared noise inherent to the videos across subjects as the data pool increases? We know that functional alignment remains a challenge because brain responses vary at the voxel level, even if the encoded information is similar across the whole region. It is unclear how, if at all, the proposed method addresses this challenge.

Secondly, the paper lacks a direct comparison with other state-of-the-art methods, which the authors partially review in the introduction. Could alternative methods achieve similar or even better performance? The authors suggest that other methods are computationally expensive in terms of parameters and training, yet they do not provide specific details. Quantitative comparisons and a more in-depth analysis of parameter costs are essential to substantiate this claim.

questions:
Please provide a detailed rationale explaining why adding more subjects is beneficial. Specifically, how does this approach address the alignment problem? A clear explanation would help clarify the method's impact on the cross-subject variability and functional alignment issues inherent to neuroAI.

Please compare the performance of the proposed method with at least two other state-of-the-art approaches. This comparison would help establish its relative effectiveness and efficiency, particularly in terms of computational costs, accuracy, and robustness across subjects.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730727298511

thread_type:
official_review

<thread_object>
creator: reviewer_dnee

<content>
summary:
This paper presents a new brain decoding paradigm for zero-shot brain decoding. Multiple experimental design exposes that brain decoding models can be generalized from existing subjects to unseen subjects.The effect of inter-subject similarity, such as gender, on generalization ability is discussed for the first time.

strengths:
1. This paper is well written and easy for the reader to read.
2. The motivation for the zero-shot setting is interesting.
3. The experimental design is very rich with multiple validations. Readers are inspired by explorations of the impact of gender.
4. In a sense, integrating the new dataset breaks the routine of NSD data use and is contributory.

weaknesses:
1. From the point of view of machine learning, the algorithms for brain decoding in this paper are basically existing methods. The use of downsampling to obtain voxels of uniform size across multiple subjects has been used, e.g. CLIP-MUSED [1] uses PCA downsampling to align different subjects, also without additional training. 
2. The methodology mentioned by the authors was not compared to any relevant studies quantitatively. The methods mentioned by the authors were not compared to any relevant studies, so it is difficult to comparatively measure whether the methods proposed by the authors are more effective or not. There are actually a subset of papers that have performed the appropriate anatomical alignment for zero-shot brain decoding[2, 3, 4]. Because of open source issues, a full reproduction may be a bit difficult, but small designs such as the use of **fsaverage** are easy to implement (fsaverage is already widely used in neuroscience). 
3. While the new integrated dataset makes sense, it seems to me that a neuroscience research task without algorithmic design requires a dataset of its own design. I think this is missing in this paper. And the decoding task is very simple, containing only the retrieval task. Although the authors mention that the method can be generalized to the reconstruction task, a glance at related papers on the same method makes it hard for me to be convinced (the same method using downsampling fails miserably on the reconstruction task).

Overall I do not deny the contribution of this paper and I think the research ideas in this paper are very interesting and deserve to be published. The reason I am currently leaning towards rejection is that I am concerned that the paper may be more suited to brain science related journals or conferences than machine learning conferences like **ICLR**. Because the methods in this paper do not involve any representation learning methods or any algorithmic innovations, and the conclusions are more of a scientific experimental nature. 

I would consider changing the rating to a positive one if the authors could convince me very well.

**Reference**
1. Zhou, Qiongyi, et al. "CLIP-MUSED: CLIP-Guided Multi-Subject Visual Neural Information Semantic Decoding." ICLR 2024.
2. Thual, Alexis, et al. "Aligning brain functions boosts the decoding of visual semantics in novel subjects." ArXiv 2023.
3. Bao, Guangyin, et al. "Wills Aligner: A Robust Multi-Subject Brain Representation Learner." Arxiv 2024.
4. Qian, Xuelin, et al. "fmri-pte: A large-scale fmri pretrained transformer encoder for multi-subject brain activity decoding." Arxiv 2023.

questions:
I don't really agree with your statement in lines 506-508 of Section 4.6 that the poor generalization of the method on the NSD dataset is due to the problem of too few subjects. It seems to me at the moment that the integrated new dataset is consistent across all visual stimuli viewed by all subjects. So even for unseen subjects, the model is trained with the images that unseen subjects 'see'. However, for the NSD dataset, the images seen by different subjects, although they are all COCO images, are not the same in the training set. Therefore, in the absence of any comparison of related methods, how can it be proved that the method proposed in this paper is better not due to the difference in dataset ?
</content>
</thread_object>

</thread>
<thread>
creation date:
1733112644122

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
A Thanks Letter and Summary

comment:
We thank all reviewers and area chairs for their valuable time, feedback, and thoughtful comments. Some comments provides valuable suggestions for improving the quality of our paper and guiding future research directions. As the ICLR reviewer-author discussion period comes to a close, we would like to provide a brief summary of the discussions.

This paper explores a novel challenge: generalizing visual brain decoding models to unseen subjects. Following the discussions, Reviewers DnEE, x87x, and QQAs have expressed positive opinions about our work, acknowledging its clear writing, detailed experiments and reasonable analyses. Reviewer DnEE remarked, “The motivation for the zero-shot setting is interesting”. Reviewer QQAs highlighted that “The analysis of the impact of similarity is interesting and original”. Reviewer x87x noted, “It has the potential to contribute meaningfully to advancing the field”. These comments reflect their support for the paper’s acceptance.

In contrast, Reviewer cDXj expressed a strong reject. We believe this stems from misunderstandings regarding both the purpose and the experimental setup of our work. To address these, we provided detailed explanations during the discussion phase. Unfortunately, Reviewer cDXj has not participated in any discussions, leaving us uncertain as to whether their concerns have been resolved. 

Once again, we sincerely thank all reviewers and area chairs for their time, effort, and valuable feedback.

Sincerely, Authors of #1426
</content>
</thread_object>

</thread>


Metareview:
metareview:
This submission contributes a method for brain-based decoding generalizing across subjects. The study is a vision task, mapping brain signals to CLIP embeddings of the images. The submission generated interest and discussion from the reviewers, in particular the across-subject generalization. The reviewers appreciated the empirical design, however the comparison to baselines is not clear at all, which is very unfortunate. The other should strive to address this last point, clarifying the corresponding points raised in the discussion and ideally running experiments to complement them.