The Wisdom of a Crowd of Brains: A Universal Brain Encoder

## Abstract

Abstract Image-to-fMRI encoding is important for both neuroscience research and practical applications.
However, such “Brain-Encoders” have been typically trained per-subject and per fMRI-dataset,
thus restricted to very limited training data.
In this paper we propose a Universal Brain-Encoder , which can be trained jointly on data from many different subjects/datasets/machines. What makes this possible is our new voxel-centric Encoder architecture, which learns a unique “voxel-embedding” per brain-voxel.
Our Encoder trains to predict the response of each brain-voxel on every image, by directly computing the cross-attention between the brain-voxel embedding and multi-level deep image features. This voxel-centric architecture allows the functional role of each brain-voxel to naturally emerge from the voxel-image cross-attention.
We show the power of this approach to (i) combine data from multiple different subjects (a “Crowd of Brains”) to improve each individual brain-encoding, (ii) quick & effective Transfer-Learning across subjects, datasets, and machines (e.g., 3-Tesla, 7-Tesla), with few training examples,
and (iii) use the learned voxel-embeddings as a powerful tool to explore brain functionality (e.g., what is encoded where in the brain).

## 1 Introduction

fMRI (*functional* MRI) has emerged as a powerful tool for measuring brain activity. This enables brain scientists to explore active brain areas during various functions and behaviors [, , , , ].
However, a human can spend only limited time inside an fMRI machine. This results in fMRI-datasets too small to span the huge space of brain functionality
or visual stimuli (natural images). Moreover, the variability in brain structure and function responses between different people [, , , ] makes it difficult to combine data across individuals that have not been exposed to the same stimuli.
All of these form severe limitations on current ability to analyze brain functionality.

Figure: Figure 1: Overview. The Universal Image-to-fMRI Brain-Encoder can train jointly on multiple subjects & datasets. It leverages cross-attention between learned shared brain-voxel embeddings and deep image features.
Refer to caption: x1.png

Image-to-fMRI encoding models, which *predict* fMRI responses to natural images, have greatly advanced the field.
With the rise of deep learning, sophisticated encoding models have emerged  [, , , , , ], offering novel insights into brain function [, , ].
However, despite these advances, these models are primarily subject-specific and machine-specific, requiring extensive individual data (which is prohibitive) for effective training.
This limits the practical use of existing brain-encoders,
and prevents their ability to leverage cross-subject data.
Attempts to create multi-subject encoders (e.g., [, , , ]) have so far been very restrictive (see Sec. 2).
These approaches have thus far not demonstrated success in merging data from multiple fMRI datasets with different stimuli
and varying acquisition settings (different machine resolutions, different scanning protocols, etc.)

In this paper, we introduce the first-ever *Universal Image-to-fMRI Brain-Encoder*, which *jointly* trains on and integrates information from a collection of very *different fMRI datasets* acquired over the years (see Fig. 1).
These multiple fMRI datasets
provide *multiple subjects* exposed to
*very different image stimuli*,
scanned on *different fMRI machines* (3-Tesla, 7-Tesla), with varying number of brain-voxels.
What makes this possible is our new Brain-Encoder architecture, which learns a unique “voxel-embedding” vector per brain-voxel
(a small cube volume within the brain).

Our Encoder trains to predict the fMRI response of each brain-voxel on each input image, by aggregating the *cross-attention* between the brain-voxel *embedding* and multi-level deep-features of the image. Other than the voxel-specific embedding
(a learned 256-dimensional vector),
all other network weights are shared across all subjects and voxels.
This strategy, of learning meaningful brain-voxel embeddings via *voxel-image cross-attention*, provides several unique benefits:
(i) The *functional role* of each brain-voxel naturally emerges.
(ii) The Brain-Encoder architecture is not restricted to a predetermined number of voxels per fMRI scan, a common limitation among existing brain encoders. This allows to train the encoder *jointly* on subjects scanned using fMRI machines with different scanning resolutions.
(iii) When a new subject/dataset is introduced, all that needs to be
learned is the new subject’s voxels embeddings. Since this is captured by a small number of weights, it can be learned with few training examples.

The *per-voxel* Embedding puts a focus on individual voxel characteristics, independent of subject identity or fMRI dataset.
This allows voxel functionality to be accurately captured across different subjects/datasets/machines.
Moreover, the cross-attention mechanism between these voxel-embeddings and *multi-level* deep image features enables each brain-voxel to appropriately align with its corresponding “semantic level” (whether low or high).

We show the power of our approach to a variety of tasks, including: (i) Integrate information from many different fMRI datasets obtained by a “Crowd of Brains”. This wealth of training data gives rise to a Universal brain encoder, whose performance/accuracy significantly exceeds that of individually-trained (subject-specific) brain-encoders. (ii) Simple *Transfer-Learning* of the Universal encoder to new subjects and new datasets, with very few training data per subject. (iii) The learned voxel-embeddings provide a new tool to explore brain functionality, providing insights into what is encoded where in the brain.

The contributions of this paper are therefore:

- •
The first-ever *Universal Brain-Encoder*, which can successfully integrate data from *multiple diverse fMRI datasets* (old & new), collected on *many different subjects*, with *different fMRI machines* (3T, 7T), on very *different image datasets*.
- •
Universal-encoding significantly improves over individually-trained *subject-specific*
encoding.
- •
*Transfer-Learning* of the Universal-Encoder to new subjects/datasets with few training data.
- •
Learn *functionally-meaningful* brain-voxel embeddings via multi-level *voxel-image cross-attention*. This powerful representation further allows to *explore brain functionality*.

## 2 Related Work

#### Visual Brain Encoders:

Visual brain encoders, which map complex visual stimuli to brain activity, have significantly advanced over the years.Visual brain encoders have significantly advanced the field of neuroscience by mapping complex visual stimuli to brain activity. Initially, these models utilized linear regression between hand-crafted image features, to predict fMRI responses on images [, ]. Over time, the field has evolved to incorporate deep learning approaches both for image feature extraction and training [, , , , , ]. These models are typically
*subject-specific*, due to substantial differences between brain responses of different people [, , , ]. This not only prevents broad generalization of these models, but also restricts their training due to the limited amout of data available per subject.

#### Multi-Subject fMRI Representations:

Traditionally, multi-subject fMRI studies have relied on *anatomical alignment* – i.e., canonical brain mapping to align all brains to a common anatomical space [, , , ].
This often results in poor *functional* correspondences (i.e., poor alignment of multi-subject fMRI responses), due to the varied nature of brain functionality across people [, , , , ].
More advanced methods for *functional alignment* were recently proposed, such as
Hyper-Alignment [, , , ]
or dimensionality reduction using auto encoders [].
However, these approaches typically require
*shared data* (i.e., same images seen by multiple subjects), which significantly restricts their applicability, and cannot be used across different (mutually-exclusive) fMRI datasets.
Our method overcomes these limitations by introducing a unique "voxel-embedding" for each brain-voxel, generated through predicting its response to images (via multi-level voxel-image *cross-attention*). This approach allows for tailored functional learning and information sharing across different subjects/datasets, without the need for any shared data or same fMRI machine.

#### Multi-Subject Brain Encoders:

Few attempts have been made to develop models that can benefit from multi-subject brain data [, , , ]. None of these approaches, however, have demonstrated the capability to integrate data across multiple subjects from different datasets, with varying fMRI resolutions, or in the absence of shared data. Methods like [] use Multi-Subject fMRI representations to integrate data from different subjects but, as mentioned above, these require some shared-data. Other approaches employ end-to-end multi-subject encoders with partially shared weights [], or use one subject’s encoder parameters as the basis for another [], or leverage pre-trained encoder outputs for new subject adaptation []. None of those approaches have demonstrated effectiveness on diverse datasets and machines. Moreover, they usually treat each fMRI scan as a single whole entity, thus preventing effective learning and sharing of *functional* knowledge across different brain-voxels (whether of the same person or different people).
In contrast, in our *voxel-centric* Universal Brain-Encoder
all network weights are shared across all brain-voxels (for all subjects/datasets/machines), *except* for the unique
*voxel-specific embedding* (learned via voxel-image cross-attention). This allows our model
to learn shared *voxel-functionality* across different subjects, datasets, and fMRI machines.

## 3 The Universal-Encoder

Our Universal Encoder facilitates joint training on data from multiple subjects across various fMRI datasets, where subjects were exposed to completely different image stimuli and scanned using fMRI machines with differing resolutions (see Fig. 1). We first provide an overview of the method, followed by a detailed explanation of the architecture and training process.

Figure: Figure 2: Universal-Encoder Architecture. Input: An image & a Voxel-ID; Output: The predicted voxel activation on that image. The model has 3 main components: (a) Feature Extraction Block – extracts multi-scale image features; (b) Learned Voxel-Embedding – captures the unique functionality of each voxel; (c) Cross-Attention Block – establishes the connection between voxel-functionality and relevant visual information.
Refer to caption: extracted/5656849/figures/UniversalEncoder2.png

### 3.1 Overview of the Approach

Our Universal-Encoder learns to predict the activation of each individual brain-voxel (a small cube volume within the brain) to each viewed image. A high-level overview of our Encoder’s main components
is provided in Fig 1
(with a detailed description in Fig 2). The model’s underlying assumptions and limitations are provided in Appendix B.

The core innovation of our encoder lies in its integration of brain data with image features through a *brain-image cross-attention* mechanism. For effective integration, we developed a per-voxel learned embedding (for each brain-voxel of each subject). Each voxel-embedding is randomly initialized, and is refined during training to accurately capture the functionality of the voxel. Given an image and a voxel embedding, the encoder extracts various image features using a DINO-v2 [] adapted model as our feature extraction block. It outputs image features from different intermediate layers of DINO, allowing each brain-voxel to attend to the appropriate semantic levels of features that align with its functionality. These image features, along with a specific voxel embedding, are processed through the cross-attention block to integrate them effectively and predict the voxel’s activation in response to the given image. Note that this architecture is *indifferent to the number of brain-voxels in each fMRI scan*, hence is applicable to any fMRI data.

Our training process optimizes all 3 components simultaneously – the voxel embedding, the feature extraction block, and the cross-attention block – with the common goal of predicting the voxel response to the input image. This joint learning framework develops meaningful voxel embeddings, that not only improve voxel response prediction, but also implicitly captures its *functional role* in the brain.
Our encoder and all associated weights are shared across all brain-voxels (for all subjects/datasets/machines), differing only in the per-voxel embeddings. This design ensures that each brain-voxel embedding is determined by its functional characteristics, rather than by its physical location in the brain or the subject’s identity.
The shared voxel embedding space supports integration of information across different voxels (whether within a single brain or across different brains). Importantly, our approach does not require subjects to have viewed the same images, nor to have been scanned in the same machine. This allows, for the first time, to integrate information from numerous fMRI datasets, collected by different groups across the globe over many years! We refer to this as the *“Wisdom of a Crowd of Brains”*.

Our proposed Universal-Encoder provides a powerful means for integrating data from multiple subjects, both within and across different fMRI datasets. These are empirically evaluated in Sec. 4. Moreover, it learns the *functional role* of each brain voxel, and maps *functionally-similar* brain voxels (both within the same brain, and across different brains) to nearby Voxel-Embeddings. This provides a powerful tool to explore the human brain and discover new functional regions within it. This is demonstrated in Sec. 5.
What facilitates such advanced brain exploration is the enormous number of images
that the large “crowd of brains” has *collectively* seen (which is prohibitive for a single subject).

### 3.2 Architecture and Training

The architecture of our encoder is designed to receive *2 inputs*: (i) an image, (ii) a Voxel-*ID*, and *outputs* a single scalar value – the predicted fMRI voxel activation on that image. The encoder architecture comprises three main components: the image features extraction block (Fig. 2a), a shared voxel embedding space (Fig. 2b), and the Voxel-Image cross-attention block (Fig. 2c).

#### Image Features Extraction Block

(Fig. 2a). This block utilizes an adapted DINO v2 model [] to derive multi-scale image features. Features are extracted from L=5 intermediate layers of the DINO V2 VIT-G/14 model (layers 1,6,12,18,24), where lower layers capture low-level image features and higher layers provide more semantic information. This hierarchical feature extraction is crucial, as voxels in the visual cortex correspond to a range of image attributes – from simple visual details to complex semantic content. Each layer’s features are projected to a lower-dimension C (using a linear layer), and are then concatenated along another dimension of length L (the number of layers). Given that DINO operates on P image patches, the final feature output is of size L×P×C. In order to transform Dino features into features that are suitable for predicting brain activity, we used a LoRA inspired approach [], that is more suitable for data-limited settings (see A.2 for details).

#### Per-Voxel Embedding

(Fig. 2b). Each brain voxel of each subject is assigned with a vector of length E=256. This E-dimensional vector (“Embedding”) is initialized randomly, and is learned during training to *maximize the prediction* of the voxel fMRI activation on images (computed via cross-attention of the voxel-embedding with image features). This voxel embedding is designed to capture the unique functionality of each voxel, facilitating the mapping of brain voxels from different subjects to the same functional space. Note that the final voxel-embedding is not image-dependant.

#### Cross-Attention Block

(Fig. 2c).
The voxel-image cross-attention block establishes the connection between voxel *functionality* and relevant visual information. This block includes three sequential components: (i) *Spatial-attention*, (ii) MLPs (multi-layer perceptrons), and (iii) *Functional-attention*. The Spatial-attention component allows the voxel embedding to select relevant locations within the image, while the Functional-attention component selects the relevant features at these locations. Both components are essential, as different brain voxels have varying image receptive fields (in both location and size – some have a small receptive field within the image, others are affected by the entire image), and different functionalities (e.g., low-level versus high-level semantic features).
Given the features of the input image (referred to as “input features” from here on),
the Spatial attention enables each voxel to focus on its corresponding spatial location within the image, effectively selecting features from the appropriate image patches. Using attention notations, the “Values” V are the input features, with dimension L×P×C. The “Query” vector $\mathbb{q}$ is the Voxel-Embedding, transformed by a linear layer which preserves its size (1×E).
The "Keys" K are derived by adding a learned per-patch positional encoding (size P×E) to the input features projected to the embedding size E.
The output is calculated by $softmax(\mathbb{q}{K_{L}}^{T})V_{L}$ for each of the L layers separately (a weighted summation across the spatial dimension P), outputting vectors of size L×C.
The spatially averaged features are then fed to MLPs (a separate 2-layered MLP for each of the L image-feature layers), maintaining the dimensions L×C.
Lastly, the Functional-attention performs a weighted summation of the
spatially-attended features to derive a single scalar voxel activation.
In this layer, $\mathbb{v}$ represents the flattened MLP output (size 1x(L*C) ), $\mathbb{q}$ is the voxel embedding itself, and K is learned feature embedding that has an entry for each of the LXC features (size (L*C)XE).
The output is calculated via $(\mathbb{q}K^{T})\mathbb{v}^{T}$. This block outputs the voxel prediction as a scalar value.

#### Training:

The model is trained end-to-end with all components learned together, where the objective of the model is to correctly predict the voxel activation on each input image.
Our training dataset comprises images along with corresponding fMRI scans, collected on many different subjects from multiple different fMRI datasets. Training batches are constructed from 32 randomly selected images, where for each image we randomly sample 5000 voxel indices along with their corresponding fMRI activations for prediction. Each subject (brain) has its own unique voxel indices.
The model is trained using the Adam optimizer with a learning rate of 1e-3. For the loss function, we employ the same loss as in [], using an affine combination of MSE loss and cosine proximity:

$$ $\mathcal{L}\left(\hat{r},r\right)=\alpha\cdot\text{MSE}\left(\hat{r},r\right)- \left(1-\alpha\right)\cos\left(\angle\left(\hat{r},r\right)\right)$ (1) $$

where $\hat{r}$ and $r$ are the *predicted* and *measured* fMRI activations, respectively, and $\alpha=0.1$.
Training the Universal-Encoder jointly on 8 NSD subjects (see ‘Datasets” below), takes $\sim$1 day on a single Quadro RTX 8000 GPU. Inference time (Image-to-fMRI encoding) takes $\sim$50 ms per-image.

Figure: Figure 3: Qualitative Evaluation of the Universal-Encoder. (a) Visual comparison of Real vs. Encoder-predicted fMRIs for 3 test images. (b) Top 5 retrieved images for each "Query" test-fMRI (see text for details).
Refer to caption: extracted/5656849/figures/retrieval_brains.png

## 4 Experiments & Evaluations

In this section we empirically evaluate our Universal-Encoder and its effectiveness for integrating data from multiple subjects/datasets. This section is organized as follows. We first present the datasets and the quantitative metrics used for evaluating the performance of the Universal Encoder.
Next (Sec. 4.1), we demonstrate the ability of our Universal-Encoder to jointly train on multiple different subjects who were never exposed to any shared data, thus exploiting the union of all their different training sets. We further show that this exceeds the performance of any individual subject in the cohort. We then show (Sec. 4.2) that old 3T fMRI datasets with limited low-resolution data can be significantly improved by leveraging a new high-quality 7T dataset. Finally (Sec. 4.3), we show that an already trained Universal-Encoder can be easily adapted to new subjects/datasets with minimal amount of new training data, using Transfer-Learning.

#### Datasets

We experimented with 3 prominent fMRI datasets: (i) The old "vim-1" dataset [, , ], which features around 1750 train and 120 test *grayscale* images , and their corresponding 4-Tesla fMRI recordings for 2 subjects. (ii) The “fMRI-on-ImageNet” dataset [], which comprises 1200 train and 50 test pairs of natural images from ImageNet [], with 3-Tesla fMRI recordings on 5 subjects. (iii) The "Natural Scenes Dataset" (NSD) [], a new 7-Tesla dataset with 8 subjects, each having around 9000 unique *subject-specific* images, and $\sim$1000 images *shared* across all subjects (which we use as our test set). This results in a total of 73,000 images, all taken from the COCO dataset []. A few example images from each of the three datasets are displayed in Figure 1.

#### Evaluation Metrics

We evaluate our Universal-Encoder (i.e., its ability to correctly predict the fMRI responses of different subjects to a variety of images),
using two quantitative measures.
Given a set of N Test images with ground-truth fMRI per subject, we first predict the fMRI responses of those N images with our Encoder. We can then compute:
*(i) Pearson Correlation* (per voxel) – We compute the normalized correlation between the sequence of $N$ *predicted* fMRI activations of each brain-voxel (on all $N$ Test images), versus the $N$ ground-truth fMRI activations of that voxel on those images.
*(ii) Image Retrieval* (per image) –
For each real fMRI scan in the test set (denoted as “Query”), we aim to retrieve (detect) its corresponding test image which produced it (out of all N test images). We first predict the fMRI from each test-image. We then measure the degree of similarity between these N *predicted* fMRIs and the *real* test fMRI scan (via normalized correlation), and sort the N predicted fMRIs accordingly. The test fMRI obtains a Retrieval Ranking score $k$, if the fMRI predicted from its real test image is located in the $k$th place (out of $N$) in the sorted list.

Figure 3 provides a *visual* (qualitative)
example of the Retrieval-Ranking test (for Subject 1 of the NSD dataset). Fig. 3a displays the real and the Encoder-predicted fMRIs for a few sample test images. As can be seen, there is a high similarity between the real fMRI scan and the Encoder-predicted one.
Fig. 3b shows the top-5 retrieved images (out of N=1000) for a few example “Query” test-fMRIs. The ground-truth test-image of each Query test-fMRI is displayed in the leftmost column of Fig. 3b.
The retrieved images are ranked by the similarity of their Encoder-predicted fMRI to the real “Query” fMRI scan of the test image. As can be seen, there are many distracting (very similar) images among the 1000 test images. Yet, our Universal-Encoder is able to obtain an *average* retrieval-rank score of 1.85 (out of 1000) for Subject 1 (averaged over all 1000 Query test-fMRIs).

Figure: Figure 4: The Wisdom of a Crowd of Brains. By aggregating data from multiple subjects, our Universal-Encoder effectively improves encoding of any individual subject. We compared 3 models: (i) The "Baseline" single-subject encoder of [], (ii) "Universal Encoder - single subject" – our architecture trained on each subject separately, (iii) "Universal Encoder - multiple subjects" – our model trained on data from 8 subjects. (a) Pearson correlation (per voxel) between predicted & ground-truth fMRI (Median value, 75th & 25th percentiles). (b) Retrieval accuracy: ranking of the GT image within the Top-1 & Top-5 for "Query" fMRI.
Refer to caption: extracted/5656849/figures/wisdom_of_crowds.png

### 4.1 The Wisdom of  a “Crowd of Brains”

We first demonstrate the Universal-Encoder’s ability to exploit data from multiple subjects, without any shared-data. For this we use the new 7-Tesla NSD dataset. Our Encoder’s train-set comprised the union of all the 8 subject-specific training sets ($\sim$9000 unique images per subject), resulting in a total of $\sim$72,000 pairs of images with their corresponding fMRI scans. Our Encoder’s test-set comprised the $\sim$1,000 shared images (the images that all 8 subjects saw).
We compared 3 models in our evaluation: (i) As a baseline, we used the image-to-fMRI encoder of [], trained separately for each subject on their subject-specific training-set (“Baseline encoder”). (ii) Our Universal-Encoder trained on each subject separately (“Universal Encoder - single subject”), and (iii) Our Universal-Encoder trained on all 8 subjects jointly, using their combined training-sets, and tested on each subject individually (“Universal Encoder - multiple subjects”).

Fig. 4a shows the *median* Pearson correlation value (along with the $25$th & $75$th percentiles – indicated by a rectangular bar around the median value), computed between the predicted activations and the ground-truth activations for all fMRI voxels. These are computed per subject, for each of the 3 models. Our Universal-Encoder, even when trained on a single subject, performs consistently better than the Baseline-encoder. Moreover, when our Universal-Encoder is trained jointly on the training sets of all subjects, it consistently outperforms all subject-specific models. It obtains notable improvements for both the “best” subjects (e.g., Subject 1 with $\sim$7% improvement) and the “worst” subject (Subject 8 with $\sim$15% improvement).

Fig. 4b further presents quantitative *Retrieval* results evaluated per subject for all 3 models. It shows the percent of times the correct image (corresponding to the "Query" fMRI) was ranked $1$st (Top-1) and among the Top-5 retrieved images.
The results indicate superior performance of the multi-subject Universal-Encoder compared to the 2 other models in both Top-1 & Top-5.
Significance testing of the Universal Encoder’s improvement over other models, as shown in Fig. S8, reveals a maximum p-value of approximately $\sim$3e-12.
The above experiments demonstrate that our Universal-Encoder can effectively aggregate data from multiple subjects (who viewed different images), while enhancing the performance of each subject individually.

### 4.2 The Wisdom of  a “Crowd of Datasets”

The Universal-Encoder can aggregate fMRI data from multiple datasets,
each with own scanning resolution and unique
image domain (e.g., B/W vs. color images). This allows training the Universal Encoder *jointly* on a high-quality (7T) and lower-quality (3T) datasets, thus significantly enhancing the encoding performance of old lower-quality datasets. Fig. 5 demonstrates this by training the encoder on the NSD dataset alongside two low-resolution datasets : VIM1 and “fMRI-on-ImageNet”,
and testing the encoding performance on individual subjects within those datasets.

Fig. 5a compares 3 models for the “fMRI-on-ImageNet” dataset: (i) The single-subject encoder of [] (“Baseline encoder”), (ii) Our Universal-Encoder trained on all subjects within “fMRI-on-ImageNet”(“Universal Encoder - same dataset”), and (iii) Universal-Encoder trained on subjects from both “fMRI-on-ImageNet” and NSD (“Universal Encoder - multi datasets”).
Our multi-subject same-dataset encoder (green) outperforms the single-subject baseline model (blue). Adding data from NSD
yields even further improvement (red). Median correlation, 75th & 25th percentiles are shown.
Fig. 5b shows results for the VIM1 dataset. Here too, adding training data from the high-resolution NSD dataset, significantly enhances encoding performance on VIM1.

Figure: Figure 5: The Wisdom of the Crowd of Datasets: Adding data from a high-quality 7-Tesla dataset (NSD) to lower-quality datasets significantly enhances encoding performance in the lower-quality datasets.
Refer to caption: x2.png

### 4.3 Transfer-Learning to New Subjects/Datasets

When a new subject/dataset is encountered, it is not necessary to train the universal-Encoder from scratch. Instead, we can add a new subject via quick Transfer-Learning, which is particularly useful when the new subject-specific data is scarce. In our transfer-learning, all weights of the pre-trained Universal-Encoder remain fixed, and *only the 256-dimensional Voxel-Embeddings* are optimized for the new subject. This allows rapid and effective transfer learning, with little new training data.

To demonstrate the effectiveness of such Transfer-Learning, we *pre-train* the Universal-Encoder
on 6 subjects from the 7-Tesla NSD dataset (Subjects 2-7). We then adapt it via transfer-learning to new subjects (without shared data) – both new subjects within NSD (Subjects 1 & 8), and to subjects in entirely different (older) fMRI datasets (“fMRI-on-ImageNet” & VIM1).
Each of the 3 plots in Fig. 6
Compares: (i)  Transfer-Learning of the *pre-trained* Universal-Encoder to the new subject, with varying numbers of subject-specific training data (purple curve), and (ii) a dedicated subject-specific model, trained from scratch on the subject-specific data only (orange curve). The x-axis represents the number of *subject-specific training examples*, and the y-axis shows the mean and standard deviation of the median Pearson
correlation between the predicted and real fMRI scan from the new subject’s test-set, over 5 runs with different random initialization & data sub-sample.

The *transferred* Universal-Encoder significantly outperforms any single-subject model on all datasets. Transfer-Learning to Subject 1 within NSD (Fig. 6a) obtains more than 77% improvement for any number of training examples. Moreover, with as little as 100 subject-specific training examples, it already achieves better results than the subject-specific model trained on the entire train-set (of 9000 training examples). A similar gap in performance is achieved for Subject 8. The transferred Universal-Encoder reaches a performance plateau at $\sim$1,000 examples.

Fig. 6b ("fMRI-on-ImageNet") & Fig. 6c (VIM1) demonstrate transfer learning from a *new* 7-Tesla dataset to *older* lower-resolution 3-Tesla or 4-Tesla datasets. These were scanned on very different types of images (e.g., VIM1 was scanned on B/W images with a circular black mask), and have much smaller train-sets.
Transfer-Learning shows a significant improvement for any number of training data, with an improvement of more than 84% for “fMRI-on-ImageNet” and $\sim$45% for VIM1 when using the entire (small) subject-specific train-set. Significance testing shown in appendix Fig. S9.

Figure: Figure 6: Transfer-Learning of Universal-Encoder to new subjects/datasets. Pre-trained on NSD, the Universal-Encoder adapts to new subjects with limited data. Transfer-Learning (purple) significantly outperforms single-subject models (orange). The plots show mean and standard deviation of the median Pearson correlation over 5 runs as a function of the number of training examples.
Refer to caption: x3.png

Figure: Figure 7: Exploring The Brain: Clustering voxel-embeddings by their proximity in the shared embedding space allows to discover and explore functionality of brain regions. The functional role of each detected cluster of voxels is understood by viewing the images that most strongly activate these clusters (see text for more details).
Refer to caption: extracted/5656849/figures/brain_clusters.png

## 5 Exploring the Brain using Voxel-Embeddings

As part of the training, our Universal-Encoder learns the *functional role* of each brain voxel. It maps *functionally-similar* brain voxels (whether within the same brain or across different brains) to nearby Voxel-Embeddings in the shared embedding space. This provides a powerful means to explore the human brain and discover new functional regions within it.
What facilitates such advanced brain exploration is the *enormous* number of images that a large “crowd of brains” has *collectively* been exposed to (which is prohibitive for any single subject).
In this section, we demonstrate that these learned voxel-embeddings, which combine data across subjects/datasets, capture semantically meaningful brain functionality and serve as robust voxel representations to explore the brain.

The standard way of dividing the brain into functional regions relies on predefined *anatomical* regions. This is a very coarse division which overlooks individual differences. We aim to find a division that is *functionally-consistent* across different subjects, while not being constrained by their anatomical differences.
To achieve this, we apply *k-means clustering* to all brain-voxel-embeddings of multiple “good” subjects (subjects with high prediction accuracy) from the 7-Tesla NSD dataset. Strong clusters often indicate the detection of a meaningful (joint) functionality within the brain.

To uncover the functional roles of such clusters, we examine which image stimuli result in fMRI scans that induce the *highest* activation levels per cluster (averaged over all voxels within the cluster). Fig. 7a displays an example of the top 4 interesting discovered clusters (displayed on the brain with different colors), along with corresponding top images (images with the highest activation per cluster) for NSD Subjects 1 & 2.
Each of these clusters has a clear and distinct functional role, being strongly activated by images of *Food, Faces, Text, Outdoor-Scenes*, respectively. These results show that our voxel-embeddings capture functional roles rather than individual identities, hence provide a potentially powerful tool to discover shared & unique brain functionalities across different people.

We further explored the ability to detect finer functional granularity within known brain regions. Fig. 7b shows one such example – the detection of functionally-meaningful clusters (sub-regions) *within* the PPA brain region (an area corresponding to places/scenes). Two clear and distinct sub-regions have emerged from our voxel-embedding clustering – Indoor-Scenes vs. Outdoor-Scenes. This demonstrates the power of our embeddings to uncover new functional regions beyond predefined anatomical boundaries. Additional examples & experiments are found in the appendix (Figs.S10, S11).

## 6 Conclusion & Discussion

This paper presents the first-ever *Universal* Image-to-fMRI Brain-Encoder, which can integrate data from many different subjects and different fMRI datasets collected over the years.
This is facilitated by a new *voxel-centric architecture*, which learns individual “voxel-embedding” per brain-voxel, via cross-attention with hierarchical image features. In this *voxel-centric* architecture, the functional role of individual brain voxels naturally emerges, leading to better encoding performance and providing a new tool to explore brain functionality (e.g., what is encoded where in the brain). Moreover, this architecture is *indifferent* to the number of brain-voxels in each fMRI scan, hence can be applied to any fMRI data.

Our approach could potentially be extended to developing Brain-Encoders for other data modalities (e.g., video, audio, speech). Moreover, the Voxel-Embeddings may potentially provide a means for exploring the existence or lack of brain functionalities in ‘irregular’ brains (such as autistic brains, or brains of visually impaired people). Since a trained Universal-Encoder can adapt to new subjects/datasets with minimal new data via Transfer-Learning, it could practically explore such new domains with only a small number of new subject-specific scans required.

## Appendix

## Appendix A Further Technical Details

### A.1 fMRI datasets

The datasets utilized in our study comprise BOLD fMRI responses to various natural images, recorded over multiple scanning sessions. Peak BOLD responses corresponding to each stimulus were estimated. Each dataset underwent unique pre-processing procedures, detailed in their respective publications [, , , ].

Two additional processing steps may be needed: voxel selection from the total of all brain voxels and per voxel normalization. For each voxel, in each run, Z-scoring normalization was performed. A ’run’ refers to one continuous period of fMRI scanning, and this normalization process standardizes the voxel responses across different runs, enhancing the comparability and consistency of the data.

#### NSD Dataset.

For the NSD dataset [], we used a post-processed dataset with voxel selection from []. They chose vision-related areas, resulting in a total of approximately 40,000 voxels, and provided voxels after per voxel normalization.

#### “fMRI-on-ImageNet" Dataset.

For the fMRI-on-ImageNet dataset [], a relevant set of around 5,000 voxels was already provided. We implemented Z-scoring normalization ourselves for this dataset.

#### VIM1 Dataset.

For the VIM1 dataset [, ], we selected the best 7,000 voxels according to the highest Signal-to-Noise Ratio (SNR). SNR is calculated as the ratio of the variance of averaged (repeated) measurements for different stimuli to the average variance of measurements for the same stimuli. This approach ensures the selection of voxels most representative of neural activity in response to diverse visual stimuli. Z-scoring normalization was also implemented for this dataset.

### A.2 modified LoRa

The Lora adaptation  [] is done by adding learned low rank matrices to weights in the original network. In the original paper the value projection and query projection weights ($W_{v}$,$W_{q}$) in the self attention block are modified. We only modify the output projection weights($W_{o}$) of the self attention block.

## Appendix B Limitations

There are two main underlying assumptions in this work that are commonly taken in most works in this area. The assumptions are that the fMRI response is memory-less and replicable. By memory-less, we mean that previous images seen by the subject do not affect the response to the current image. We wanted our model to be as general as possible and applicable to many datasets; adding memory dependence would hinder this. By replicable, we mean that the same response for an image will be measured regardless of when the subject sees the image. This is important when averaging multiple responses to the same image, a general practice in the field due to the low SNR of the fMRI signal.

Moreover, it is important to note that there is significant variability in measured signal quality across subjects. The brain exploration we demonstrate is done for subjects with relatively high SNR. For subjects with poor signal quality, it is harder to obtain a good estimation of voxel functionality, and we would likely not achieve as good results for meaningful segmentation of brain regions.

## Appendix C Additional Figures

Figure: Figure S8: Retrieval Results: Our universal encoder trained on multiple subjects significantly improves retrieval accuracy over both subject-specific models as evidenced by the calculated p-value. We conducted 10 retrieval experiments, wherein we randomly sampled 999 different distracted images for each “query” fMRI. Results are compared across three models: (i) "Baseline Encoder" – the encoder of [] trained separately on each subject, (ii) "Universal Encoder - Single Subject" – our architecture trained separately on each subject, and (iii) "Universal Encoder - Multiple Subjects" – our model trained on combined data from 8 subjects. In (a) we present the mean top-1 accuracy (across the 10 experiments) along with the standard deviation for each model and subject. (b) depicts the mean top-5 accuracy along with the standard deviation for each model and subject. (c) Table of p-values per subject illustrates the improvement significance of our universal encoder trained on multiple subjects compared to the other two models, emphasizing the significance of the improvement.
Refer to caption: extracted/5656849/figures/Sup_pvalue_retrieval.png

Figure: Figure S9: Transfer Learning Significance Results: The transferred Universal-Encoder significantly outperforms any single-subject model on all datasets. This figure shows the p-value of the median Pearson correlation between the predicted and real fMRI scan for each dataset across 5 runs. For each dataset we compared: (i) Transfer-Learning of the pre-trained Universal-Encoder to the new subject, with varying numbers of subject-specific training data, and (ii) a dedicated subject-specific model, trained from scratch on the subject-specific data only. This extends the findings presented in Fig. 6.
Refer to caption: extracted/5656849/figures/Sup_transfer_pvalues.png

Figure: Figure S10: Voxel Embedding Clusters: 5-top images which activated voxels within each cluster. Most clusters are consistent across subjects indicating that voxel embeddings capture functional roles rather than individual identities.
Refer to caption: extracted/5656849/figures/embedding_edited_predicted.png

Figure: Figure S11: Exploring the brain: Clustering voxel-embeddings by their proximity in the shared embedding space allows to discover and explore functionality of brain regions. As an example, within the EBA brain region (an area corresponding to body parts), it identified functionally-meaningful clusters, revealing three distinct sub-regions. The functional role of each detected cluster of voxels is understood by viewing the images that most strongly activate these clusters, in this case: images of sports, a crowd of people and food.
Refer to caption: extracted/5656849/figures/Supp_EBA_clusters.png

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: What is the spatial resolution of the pre-processed fMRI datasets and the corresponding dimensionality of the 4D volumetric data? It is curious whether the spatial resolution can support the fine-grained analysis of brain response as shown in Fig. S13.
soundness: 2
strengths: 1. The proposed Universal Brain-Encoder can effectively handling sequences from different subjects, datasets, and machines, which enhances its applicability for both neuroscience research and practical applications

2. The paper presents comprehensive experimental results, and the proposed Universal Brain-Encoder achieves satisfied performance across multiple datasets. Notably, it achieves substantial performance improvements when trained on multi-dataset inputs, supporting the authors' argument regarding the "Crowd of Brains" concept
summary: This paper introduces a Universal fMRI Encoder for the prediction of brain responses to image stimuli. Unlike traditional subject-specific brain encoding models, the proposed work is trained and validated across multiple subjects and datasets. The model learns voxel embedding through cross-attention with multi-level deep image features, allowing the model to capture functional roles of different brain regions while sharing other network weights across subjects. The model is evaluated on 3 datasets on two measurements: a comparison of the estimated fMRI signal vs. ground truth and the image retrieval accuracy using top-k accuracy.
weaknesses: 1. The idea appears to closely resemble existing works such as [1], MindFormer [2], MindEye2 [3], MindBridge [4], and BDI [5]. These studies also learn a set of independent parameters for each subject while sharing most parameters across subjects. The novelty of the proposed idea needs further clarification.

2. Some brain decoding methods employ symmetric architectures, so they have both Image-to-fMRI and fMRI-to-Image networks, such as [6] and [7]. A discussion about these approaches should be included in the comparative experiments.

3. The quality of the generated fMRI data requires more validation. The authors should use additional metrics or evaluation methods to assess whether the generated data can still be used to analyze brain activity. For instance, the authors can use existing brain decoding models to prove they can reconstruct images from the generated fMRI sequences.

4. An ablation study on different Voxel Embedding dimensions should be included.

5. Is it better to utilize all voxels in fMRI sequences? The proposed voxel-based approach has the potential to capture latent semantic relationships between brain activities and input signals, whereas manually selected ROIs may lead to information loss. If the method can effectively model all voxels and provide visualize results as demonstrated in Fig. 7, it would yield interesting results.

[1] Functional Brain-to-Brain Transformation with No Shared Data
[2] MindFormer: A Transformer Architecture for Multi-Subject Brain Decoding via fMRI
[3] MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data
[4] MindBridge: A Cross-Subject Brain Decoding Framework
[5] Brain Dialogue Interface (BDI): A User-Friendly  fMRI Model for Interactive Brain Decoding
[6] From voxels to pixels and back: Self-supervision in natural-image reconstruction from fMRI
[7] Rethinking Visual Reconstruction: Experience-Based Content Completion Guided by Visual Cues

---

confidence: 3
contribution: 4
initial_rating: 8
presentation: 4
questions: In other papers that use NSD, subject 5 is typically the best performing subject. However in this work the retrieval accuracy is quite a bit lower than subjects 1 and 2. Any ideas why this might be the case?
soundness: 4
strengths: - This is a very strong and well written paper. The methods are easy to understand and well motivated. I could see this encoder being used a lot when working with smaller vision datasets. 
- Retrieval accuracy is impressively high. It looks close to 95% top-1 accuracy for subjects 1 and 2 across 1000 test images (chance is 0.1%).
- Statistical tests are performed for all experiments.
summary: This work trains a universal brain encoder that can predict the brain responses from multiple participants and datasets. The input to the encoder is a stimulus image and a learned 256-dimensional voxel-wise embedding. The voxel-wise embeddings are randomly initialized, and contain no information about the participant or the spatial location of the voxel. All other parameters in the encoder are shared between voxels, participants, and datasets.

- The predictions are evaluated with per-voxel pearson correlation and a per-image retrieval metric.
- Their universal encoder significantly outperforms baseline single-subject encoders (figure 4)
- Inclusion of a higher quality 7T data improves performance on older 3T and 4T datasets (figure 5)
- A pre-trained encoder can transfer to new subjects and datasets just by learning new voxel embeddings. Performance is much higher and learning is faster than a single-subject encoder.
- K-means clustering is applied to the voxel embeddings to identify regions for food, words, faces, sports, indoor scenes, outdoor scenes.
weaknesses: I think the paper is lacking some exploration and visualization of the voxel embeddings. Here are some ideas:
- Apply the clustering to more than 2 participants. 
- Other clustering methods besides k-means (i.e. some that can deal with outliers)
- A flatmap visualization with outlines of previously identified category selective regions for faces, bodies, places, and words. This would be helpful for comparing to the clusters identified with k-means.
- A UMAP or tsne applied to the combined embeddings for the 8 participants, and then visualized on the cortical surface with a color mapping.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: See weaknesses.
soundness: 2
strengths: 1. The problem of the universal brain-image generation is important.

2. Experiments successfully demonstrated the proposed method can train on multiple subjects from different datasets and achieved a better performance.

3. The presentation of motivations, methods, and experiments is clear and easy to follow.
summary: This work proposed a Universal Brain Encoder that can train on multiple subjects from different datasets. The methods applied cross-attention between features of the image and fMRI voxel. Experiments on three datasets have shown that performance can be improved after fine-tuning with new subjects.
weaknesses: 1. The announcement of the *first-ever Universal Brain-Encoder* is too aggressive. The idea of the model is to be able to train on multiple subjects and datasets instead of **universally** applying to any unseen subjects or datasets. The performance of the proposed model on a new subject is tested via few-shot transfer learning instead of zero-shot learning. 

2. The method of cross-attention is not novel and exists in the field of brain-image generation [1,2,3].

[1] Sun, Jingyuan, et al. "Contrast, attend and diffuse to decode high-resolution images from brain activities." Advances in Neural Information Processing Systems 36 (2024).

[2] Wang, Shizun, et al. "Mindbridge: A cross-subject brain decoding framework." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.

[3] Scotti, Paul S., et al. "MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data." arXiv preprint arXiv:2403.11207 (2024).

3. Lack of ablation studies. For example, in Figure 7, functional embedding is not evaluated to show the functional roles of voxels but the k-means results of voxel embeddings were used. Also, other components, which were claimed essential in the main text, are not evaluated.

4. The potential power of finding subject-specific brain parcellation is interesting, but the demonstration in Figure 7 shows this can only proceed on visual networks instead of the whole brain. Brain parcellation is for the whole brain.

---

confidence: 4
contribution: 3
initial_rating: 8
presentation: 4
questions: + In Figure 2, the dimension of Voxel Embedding is "1xE", why 1 and not the number of voxels? If it's 1, does it mean that we need to train a model for each voxel? If it is the number of voxels, then the number of voxel embeddings should be much larger than the image tokens (i.e. P), and at this point does the attention module make the randomly initialized voxel embeddings overly self-concerned?
+ In Figure 6(a), why is the performance of coding saturated when the few-shot samples exceed 3000?

I'm willing to further raise my rating according to the author's rebuttal.
soundness: 3
strengths: + This paper focuses on the intersting but few-studied issue of multi-subject fMRI encoding.
+ The motivation for this paper is clear and the proposed method has yielded promising results.
+ The proposed method achieves cross-device and cross-subject few-shot transfer learning, making it highly applicable.
+ Neuroscience exploration using the voxel-wise embedding proposed in the paper is promising.
+ The paper gives detailed implementation details of the model model which helps in understanding and also ensures reproducibility.
summary: This paper proposes a multi-subject fMRI encoding model. The authors set learnable voxel-wise embedding for each subject and optimize the subject-specific voxel-wise embedding and the subject-shared encoding model through the fMRI prediction task. Through the evaluation on multiple fMRI datasets, the authors validate the effectiveness of the proposed method and further validate that few-shot cross-subject transfer can be achieved. Finally, this paper utilizes learned voxel-wise embedding to initially explore concept-selective in the brain cortex, showing its value in neuroscience applications.
weaknesses: #### 1. More related works should be discussed:
The authors should discuss additional related works or acknowledge that these related works have inspired them, including but not limited to:
+ At the level of model design, the proposed method seems to be a revised version of [1], which employs ROI-wise embedding rather than voxel-wise embedding. 
+ At the level of research ideas, some works also train encoding models and then use them for neuroscience explorations, such as [2][3].
+ at the level of fMRI representation learning, [4][5] already show the use of multi-subject data can enhance each subject's representation, and [4][6][7] already show the use of other subject's fMRI can achieve few-shot transfer learning.

#### 2. Limited evaluation metrics:

In this paper, only voxel-wise Pearson coefficients and retrieval results are used as evaluation metrics, and the inclusion of more metrics such as $R^2$, MSE, etc. can further indicate the fMRI encoding accuracy.


#### 3. On fMRI replicable

The method proposed by the authors fails to address the issue of fMRI replicability, which is a common problem with regression-based fMRI encoding models. The authors already discuss this in their limitation and assume that the fMRI captured by subjects viewing the same image multiple times is the same. However, this assumption may greatly limit the training of fMRI encoding models.



[1] Hossein Adeli et al. Predicting brain activity using Transformers. bioRxiv, 2023: 2023.08. 02.551743. 

[2] Andrew Luo et al. Brain Diffusion for Visual Exploration: Cortical Discovery using Large Scale Generative Models. NeurIPS 2023.

[3] Andrew Luo et al. BrainSCUBA: Fine-Grained Natural Language Captions of Visual Cortex Selectivity. ICLR 2024.

[4] Shizun Wang et al. A cross-subject brain decoding framework. CVPR 2024.

[5] Guangyin Bao et al. Wills Aligner: A Robust Multi-Subject Brain Representation Learner. arXiv:2404.13282.

[6] Paul S. Scotti et al. MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data. ICML 2024.

[7] Zixuan Gong et al. MindTuner: Cross-Subject Visual Decoding with Visual Fingerprint and Semantic Correction. arXiv:2404.12630.

Meta-review: This manuscript proposes a model for the prediction of volumes of fMRI (BOLD) activations given images, presumably image stimuli. It claims generalization ("universal" encoder) allowing for predictions to be made in a variety of settings in a variety of subjects. The authors demonstrate use empirically on three open datasets (`vim-1`, `fMRI-on-ImageNet`, and `NSD`).

Overall the concept of encoding and decoding for brain signals (of a variety of modalities) is not novel, and the authors accordingly provide a bevy of references. The reviewers contend that additional references may be considered as baselines or related, and the authors do not compare to them; rebuttal by the authors states that 1) the models given are mostly full brain encoders instead of voxel/vertex-wise, and 2) the listed are mostly decoder models. The authors instead compare only to the encoder branch of Gaziv et al. 2022 (NeuroImage), which is listed as "Baseline".

It seems unclear to me what the contribution of this particular manuscript provides over, e.g., the similar Gaziv et al. 2022 paper. I tend to agree with reviewer `Z1zy`, this manuscript appears to resemble previous work in form, and only provides parameterization to inter-subject variation. Moreover, it appears to be a straightforward application of transfer learning with respect to the population variation.

It occurs to me that in reading this paper, I feel that I have not learned anything extraordinary since reading the Gaziv et al. 2022 paper nor its immediate predecessor Beliy et al. 2019. This is not to say there haven't been improvements, or that the individual works aren't themselves of value, but that in re-reading all three in immediate succession during the review of `Z1zy`'s comments and its replies by the authors, I find low marginal information gain on the later papers conditional on the earlier publications. It seems that (deep) fMRI-Image decoder/encoder pairings are becoming well explored, and I think that producing work that either educates us further about the brain or about models themselves is required for truly outstanding work.

I therefore recommend rejection, but I note to the SACs/PC Chairs that this paper is quite borderline, and I am happy to be overruled. The manuscript is in acceptable form, but appears as a derivative work from transfer learning on previous methods.

Decision: Reject