The Wisdom of a Crowd of Brains: A Universal Brain Encoder

Authors: Roman Beliy, navve wasserman, Amit Zalcher, michal Irani
Primary area: applications to neuroscience & cognitive science
Keywords: Image-to-fMRI encoding, Explore the brain using ML, Brain-Image cross-attention, Visual Perception, Brain Mapping, Neuroscience, Computer Vision

Abstract:
Image-to-fMRI encoding is important for both neuroscience research and practical applications. However, such “Brain-Encoders” have been typically trained per-subject and per fMRI-dataset, thus restricted to very limited training data. In this paper we propose a Universal Brain-Encoder, which can be trained jointly on data from many different subjects/datasets/machines. What makes this possible is our new voxel-centric Encoder architecture, which learns a unique “voxel-embedding” per brain-voxel. Our Encoder trains to predict the response of each brain-voxel on every image, by directly computing the cross-attention between the brain-voxel embedding and multi-level deep image features. This voxel-centric architecture allows the functional role of each brain-voxel to naturally emerge from
the voxel-image cross-attention. We show the power of this approach to: (i) combine data from multiple different subjects (a “Crowd of Brains”) to improve each individual brain-encoding, (ii) quick & effective Transfer-Learning across sub- jects, datasets, and machines (e.g., 3-Tesla, 7-Tesla), with few training examples, and (iii) we show the potential power of the learned voxel-embeddings to explore brain functionality (e.g., what is encoded where in the brain).

Decision: reject
Reviewer scores: 3, 8, 6, 8

Reviews and discussion:
<thread>
creation date:
1730283073570

thread_type:
official_review

<thread_object>
creator: reviewer_udpx

<content>
summary:
This paper proposes a multi-subject fMRI encoding model. The authors set learnable voxel-wise embedding for each subject and optimize the subject-specific voxel-wise embedding and the subject-shared encoding model through the fMRI prediction task. Through the evaluation on multiple fMRI datasets, the authors validate the effectiveness of the proposed method and further validate that few-shot cross-subject transfer can be achieved. Finally, this paper utilizes learned voxel-wise embedding to initially explore concept-selective in the brain cortex, showing its value in neuroscience applications.

strengths:
+ This paper focuses on the intersting but few-studied issue of multi-subject fMRI encoding.
+ The motivation for this paper is clear and the proposed method has yielded promising results.
+ The proposed method achieves cross-device and cross-subject few-shot transfer learning, making it highly applicable.
+ Neuroscience exploration using the voxel-wise embedding proposed in the paper is promising.
+ The paper gives detailed implementation details of the model model which helps in understanding and also ensures reproducibility.

weaknesses:
#### 1. More related works should be discussed:
The authors should discuss additional related works or acknowledge that these related works have inspired them, including but not limited to:
+ At the level of model design, the proposed method seems to be a revised version of [1], which employs ROI-wise embedding rather than voxel-wise embedding. 
+ At the level of research ideas, some works also train encoding models and then use them for neuroscience explorations, such as [2][3].
+ at the level of fMRI representation learning, [4][5] already show the use of multi-subject data can enhance each subject's representation, and [4][6][7] already show the use of other subject's fMRI can achieve few-shot transfer learning.

#### 2. Limited evaluation metrics:

In this paper, only voxel-wise Pearson coefficients and retrieval results are used as evaluation metrics, and the inclusion of more metrics such as $R^2$, MSE, etc. can further indicate the fMRI encoding accuracy.


#### 3. On fMRI replicable

The method proposed by the authors fails to address the issue of fMRI replicability, which is a common problem with regression-based fMRI encoding models. The authors already discuss this in their limitation and assume that the fMRI captured by subjects viewing the same image multiple times is the same. However, this assumption may greatly limit the training of fMRI encoding models.



[1] Hossein Adeli et al. Predicting brain activity using Transformers. bioRxiv, 2023: 2023.08. 02.551743. 

[2] Andrew Luo et al. Brain Diffusion for Visual Exploration: Cortical Discovery using Large Scale Generative Models. NeurIPS 2023.

[3] Andrew Luo et al. BrainSCUBA: Fine-Grained Natural Language Captions of Visual Cortex Selectivity. ICLR 2024.

[4] Shizun Wang et al. A cross-subject brain decoding framework. CVPR 2024.

[5] Guangyin Bao et al. Wills Aligner: A Robust Multi-Subject Brain Representation Learner. arXiv:2404.13282.

[6] Paul S. Scotti et al. MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data. ICML 2024.

[7] Zixuan Gong et al. MindTuner: Cross-Subject Visual Decoding with Visual Fingerprint and Semantic Correction. arXiv:2404.12630.

questions:
+ In Figure 2, the dimension of Voxel Embedding is "1xE", why 1 and not the number of voxels? If it's 1, does it mean that we need to train a model for each voxel? If it is the number of voxels, then the number of voxel embeddings should be much larger than the image tokens (i.e. P), and at this point does the attention module make the randomly initialized voxel embeddings overly self-concerned?
+ In Figure 6(a), why is the performance of coding saturated when the few-shot samples exceed 3000?

I'm willing to further raise my rating according to the author's rebuttal.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730657536748

thread_type:
official_review

<thread_object>
creator: reviewer_bqd5

<content>
summary:
This work proposed a Universal Brain Encoder that can train on multiple subjects from different datasets. The methods applied cross-attention between features of the image and fMRI voxel. Experiments on three datasets have shown that performance can be improved after fine-tuning with new subjects.

strengths:
1. The problem of the universal brain-image generation is important.

2. Experiments successfully demonstrated the proposed method can train on multiple subjects from different datasets and achieved a better performance.

3. The presentation of motivations, methods, and experiments is clear and easy to follow.

weaknesses:
1. The announcement of the *first-ever Universal Brain-Encoder* is too aggressive. The idea of the model is to be able to train on multiple subjects and datasets instead of **universally** applying to any unseen subjects or datasets. The performance of the proposed model on a new subject is tested via few-shot transfer learning instead of zero-shot learning. 

2. The method of cross-attention is not novel and exists in the field of brain-image generation [1,2,3].

[1] Sun, Jingyuan, et al. "Contrast, attend and diffuse to decode high-resolution images from brain activities." Advances in Neural Information Processing Systems 36 (2024).

[2] Wang, Shizun, et al. "Mindbridge: A cross-subject brain decoding framework." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[3] Scotti, Paul S., et al. "MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data." arXiv preprint arXiv:2403.11207 (2024).

3. Lack of ablation studies. For example, in Figure 7, functional embedding is not evaluated to show the functional roles of voxels but the k-means results of voxel embeddings were used. Also, other components, which were claimed essential in the main text, are not evaluated.

4. The potential power of finding subject-specific brain parcellation is interesting, but the demonstration in Figure 7 shows this can only proceed on visual networks instead of the whole brain. Brain parcellation is for the whole brain.

questions:
See weaknesses.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730710427582

thread_type:
official_review

<thread_object>
creator: reviewer_r8mq

<content>
summary:
This work trains a universal brain encoder that can predict the brain responses from multiple participants and datasets. The input to the encoder is a stimulus image and a learned 256-dimensional voxel-wise embedding. The voxel-wise embeddings are randomly initialized, and contain no information about the participant or the spatial location of the voxel. All other parameters in the encoder are shared between voxels, participants, and datasets.

- The predictions are evaluated with per-voxel pearson correlation and a per-image retrieval metric.
- Their universal encoder significantly outperforms baseline single-subject encoders (figure 4)
- Inclusion of a higher quality 7T data improves performance on older 3T and 4T datasets (figure 5)
- A pre-trained encoder can transfer to new subjects and datasets just by learning new voxel embeddings. Performance is much higher and learning is faster than a single-subject encoder.
- K-means clustering is applied to the voxel embeddings to identify regions for food, words, faces, sports, indoor scenes, outdoor scenes.

strengths:
- This is a very strong and well written paper. The methods are easy to understand and well motivated. I could see this encoder being used a lot when working with smaller vision datasets. 
- Retrieval accuracy is impressively high. It looks close to 95% top-1 accuracy for subjects 1 and 2 across 1000 test images (chance is 0.1%).
- Statistical tests are performed for all experiments.

weaknesses:
I think the paper is lacking some exploration and visualization of the voxel embeddings. Here are some ideas:
- Apply the clustering to more than 2 participants. 
- Other clustering methods besides k-means (i.e. some that can deal with outliers)
- A flatmap visualization with outlines of previously identified category selective regions for faces, bodies, places, and words. This would be helpful for comparing to the clusters identified with k-means.
- A UMAP or tsne applied to the combined embeddings for the 8 participants, and then visualized on the cortical surface with a color mapping.

questions:
In other papers that use NSD, subject 5 is typically the best performing subject. However in this work the retrieval accuracy is quite a bit lower than subjects 1 and 2. Any ideas why this might be the case?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730913364871

thread_type:
official_review

<thread_object>
creator: reviewer_z1zy

<content>
summary:
This paper introduces a Universal fMRI Encoder for the prediction of brain responses to image stimuli. Unlike traditional subject-specific brain encoding models, the proposed work is trained and validated across multiple subjects and datasets. The model learns voxel embedding through cross-attention with multi-level deep image features, allowing the model to capture functional roles of different brain regions while sharing other network weights across subjects. The model is evaluated on 3 datasets on two measurements: a comparison of the estimated fMRI signal vs. ground truth and the image retrieval accuracy using top-k accuracy.

strengths:
1. The proposed Universal Brain-Encoder can effectively handling sequences from different subjects, datasets, and machines, which enhances its applicability for both neuroscience research and practical applications

2. The paper presents comprehensive experimental results, and the proposed Universal Brain-Encoder achieves satisfied performance across multiple datasets. Notably, it achieves substantial performance improvements when trained on multi-dataset inputs, supporting the authors' argument regarding the "Crowd of Brains" concept

weaknesses:
1. The idea appears to closely resemble existing works such as [1], MindFormer [2], MindEye2 [3], MindBridge [4], and BDI [5]. These studies also learn a set of independent parameters for each subject while sharing most parameters across subjects. The novelty of the proposed idea needs further clarification.

2. Some brain decoding methods employ symmetric architectures, so they have both Image-to-fMRI and fMRI-to-Image networks, such as [6] and [7]. A discussion about these approaches should be included in the comparative experiments.

3. The quality of the generated fMRI data requires more validation. The authors should use additional metrics or evaluation methods to assess whether the generated data can still be used to analyze brain activity. For instance, the authors can use existing brain decoding models to prove they can reconstruct images from the generated fMRI sequences.

4. An ablation study on different Voxel Embedding dimensions should be included.

5. Is it better to utilize all voxels in fMRI sequences? The proposed voxel-based approach has the potential to capture latent semantic relationships between brain activities and input signals, whereas manually selected ROIs may lead to information loss. If the method can effectively model all voxels and provide visualize results as demonstrated in Fig. 7, it would yield interesting results.

[1] Functional Brain-to-Brain Transformation with No Shared Data
[2] MindFormer: A Transformer Architecture for Multi-Subject Brain Decoding via fMRI
[3] MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data
[4] MindBridge: A Cross-Subject Brain Decoding Framework
[5] Brain Dialogue Interface (BDI): A User-Friendly  fMRI Model for Interactive Brain Decoding
[6] From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI
[7] Rethinking Visual Reconstruction: Experience-Based Content Completion Guided by Visual Cues

questions:
What is the spatial resolution of the pre-processed fMRI datasets and the corresponding dimensionality of the 4D volumetric data? It is curious whether the spatial resolution can support the fine-grained analysis of brain response as shown in Fig. S13.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731470294711

thread_type:
public_comment

<thread_object>
creator: ~zhibo_tian1

<content>
title:
Mapping Voxel Embeddings to Brain Regions.

comment:
Dear Authors,

I have a question regarding the mapping of voxel embeddings to brain regions. Given that each voxel embedding is a 256-dimensional vector, could you please clarify how these embeddings are associated with specific brain regions, particularly in Section 5, "Exploring the Brain Using Voxel-Embeddings"?

Thank you.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732112726329

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We would like to thank all the reviewers for their time, effort and valuable comments. We did our best to address all the concerns and the extra requested evaluations. The new results, evaluations and experiments we performed per the reviewers’ suggestions are now added to the Revised Appendix of the uploaded version. 

Please note that the main paper itself has not been revised yet (due to the time limit). Pointers to these new experiments from the main paper will be added in the final version of the paper, as well as the added clarifications below.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732787974795

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
Thank you for all your time, valuable comments, and suggestions. We will incorporate all the new experimental results we reported in our rebuttal (per the reviewers‘ requests), as well as the missing citations and our textual clarifications (following the discussion with the reviewers), into the final version of our paper.  This will undoubtedly make our paper clearer to future readers. 😊

We will add all these new additions either in the body of the paper or in the appendix (pending on the page-limit constraints of the main body). If added to the appendix, we will add a clear and proper reference to its location in the appendix from the paper itself.

 
Thank you for all your feedback!
</content>
</thread_object>

</thread>


Metareview:
metareview:
This manuscript proposes a model for the prediction of volumes of fMRI (BOLD) activations given images, presumably image stimuli. It claims generalization ("universal" encoder) allowing for predictions to be made in a variety of settings in a variety of subjects. The authors demonstrate use empirically on three open datasets (`vim-1`, `fMRI-on-ImageNet`, and `NSD`).

Overall the concept of encoding and decoding for brain signals (of a variety of modalities) is not novel, and the authors accordingly provide a bevy of references. The reviewers contend that additional references may be considered as baselines or related, and the authors do not compare to them; rebuttal by the authors states that 1) the models given are mostly full brain encoders instead of voxel/vertex-wise, and 2) the listed are mostly decoder models. The authors instead compare only to the encoder branch of Gaziv et al. 2022 (NeuroImage), which is listed as "Baseline".

It seems unclear to me what the contribution of this particular manuscript provides over, e.g., the similar Gaziv et al. 2022 paper. I tend to agree with reviewer `Z1zy`, this manuscript appears to resemble previous work in form, and only provides parameterization to inter-subject variation. Moreover, it appears to be a straightforward application of transfer learning with respect to the population variation.

It occurs to me that in reading this paper, I feel that I have not learned anything extraordinary since reading the Gaziv et al. 2022 paper nor its immediate predecessor Beliy et al. 2019. This is not to say there haven't been improvements, or that the individual works aren't themselves of value, but that in re-reading all three in immediate succession during the review of `Z1zy`'s comments and its replies by the authors, I find low marginal information gain on the later papers conditional on the earlier publications. It seems that (deep) fMRI-Image decoder/encoder pairings are becoming well explored, and I think that producing work that either educates us further about the brain or about models themselves is required for truly outstanding work.

I therefore recommend rejection, but I note to the SACs/PC Chairs that this paper is quite borderline, and I am happy to be overruled. The manuscript is in acceptable form, but appears as a derivative work from transfer learning on previous methods.