# UMBRAE: Unified Multimodal Brain Decoding

Weihao Xia, Raoul de Charette, Cengiz Öztireli, Jing-Hao Xue

► To cite this version:

Weihao Xia, Raoul de Charette, Cengiz Öztireli, Jing-Hao Xue. UMBRAE: Unified Multimodal Brain Decoding. ECCV 2024 - European Conference on Computer Vision, Sep 2024, Milan, Italy. hal-04986146

HAL Id: hal-04986146

https://inria.hal.science/hal-04986146v1

Submitted on 11 Mar 2025

HAL is a multi-disciplinary open access
archive for the deposit and dissemination of sci-
entific research documents, whether they are pub-
lished or not. The documents may come from
teaching and research institutions in France or
abroad, or from public or private research centers.

L'archive ouverte pluridisciplinaire HAL, est
destinée au dépôt et à la diffusion de documents
scientifiques de niveau recherche, publiés ou non,
émanant des établissements d'enseignement et de
recherche français ou étrangers, des laboratoires
publics ou privés.

<figure><img src="image_1.png" /></figure>

Distributed under a Creative Commons Attribution 4.0 International License
arXiv:2404.07202v2 [cs.CV] 18 Jul 2024

# UMBRAE: Unified Multimodal Brain Decoding

Weihao Xia¹ Raoul de Charette² Cengiz Oztireli³ Jing-Hao Xue¹

¹University College London ²Inria ³University of Cambridge

**Abstract.** We address prevailing challenges of the brain-powered research, departing from the observation that the literature hardly recover accurate spatial information and require subject-specific models. To address these challenges, we propose UMBRAE, a unified multimodal decoding of brain signals. First, to extract instance-level conceptual and spatial details from neural signals, we introduce an efficient universal brain encoder for multimodal-brain alignment and recover object descriptions at multiple levels of granularity from subsequent multimodal large language model (MLLM). Second, we introduce a cross-subject training strategy mapping subject-specific features to a common feature space. This allows a model to be trained on multiple subjects without extra resources, even yielding superior results compared to subject-specific models. Further, we demonstrate this supports weakly-supervised adaptation to new subjects, with only a fraction of the total training data. Experiments demonstrate that UMBRAE not only achieves superior results in the newly introduced tasks but also outperforms methods in well established tasks. To assess our method, we construct and share with the community a comprehensive brain understanding benchmark BrainHub. Our code and benchmark are available at https://weihaox.github.io/UMBRAE.

**Keywords:** Multimodal Brain Decoding · Universal Brain Encoder · Cross-Subject Training · Weakly-Supervised Adaptation

<figure><img src="image_2.png" /><figcaption>**Fig. 1:** Multimodal Decoding. By aligning brain features with MLLMs, UMBRAE decodes multimodal cues from brain signals, which allows multiple downstream tasks.</figcaption></figure>

## 1 Introduction

Typically, artificial intelligence research relies on intermediate modalities to interpret human intentions, such as language [9, 48], gaze [1, 54], facial expression [12], and action [19]. These modalities, however, are indirect channels of communication with humans and may be highly inaccurate for people with cognitive or physical disabilities or even locked-in patients who are conscious but unable to
communicate through speech, limb, or facial movements [24]. In this context, the potential for direct interpretation of neural signals stands out as a promising prospect. The brain imaging literature has recently advanced, decoding neural signals into various forms such as image [41], video [10], or text [47] to read intentions [8, 25]. This deepens our understanding of the brain which neural activity is not directly comprehensible to humans.

However, there are remaining challenges in brain-powered research. First, decoding brain signals into a single modality results in a lossy representation of the brain activities. On the one hand, text fails to preserve the peculiar appearance of a texture or the spatial location of an object. On the other hand, visual decoding [34, 41, 53] addresses the underdetermined problem of pixel-wise reconstruction and lacks explicitation of the scene structure. Consider, for instance, the scenario where a person uses thoughts to control a robotic arm to retrieve an apple from a fruit bowl on a table. The first task is to recognize the apple amidst similar visual concepts and then locate its exact position. But current methods lack such fine-grained decoding capability to interpret object categories, visual concepts, and their relationships. The second challenge pertains to the subject-specific patterns of brain activities [2]. Therefore, current methods typically train a model for each subject to cope with distinctive brain patterns. Decoding brain signals across multiple subjects presents challenges due to the structural and functional differences among individual brains.

Hence, we instead propose to decode a robust multimodal representation which serves as proxy for downstream tasks, such as textual or visual decoding. Our method allows brain decoding at different granularities, through prompting, which unravels unprecedented brain-machine interface for locked-in patients [24] that typically requires iterative feedback. To evaluate our novel tasks, we extend the popular Natural Scenes Dataset (NSD) [2] with multimodal ground truth, which constitutes a new brain understanding benchmark. Both code and benchmark will be made publicly available. Our contributions summarize as follows:

- We introduce UMBRAE, aiming at unified multimodal brain decoding. Our method relies on a universal brain encoder and a frozen multimodal large language model seeking to align brain signals with images. We also propose a cross-subject training strategy to learn a universal representation across subjects, as opposed to the standard subject-specific training. Furthermore, it allows the novel weakly-supervised adaptation, enabling the training of a model for any arbitrary subject with minimal training data.

- We construct BrainHub, a multimodal brain understanding benchmark extending NSD [2]. The benchmark pairs fMRI with semantic concepts and spatial localization in visual stimuli, offering tasks and metrics for evaluation.

- Our method achieves better or on par performance compared to state-of-the-art methods on a variety of tasks including brain captioning, retrieval, and visual decoding. It is also the first one to enable direct brain grounding, performing on par with natural baselines while being at least 10 times faster.
## 2 Related Works

**Brain-Conditioned Generation.** Generative vision models conditioned on brain signals [28, 34, 41, 44, 53] have recently achieved unparalleled performance in decoding visual stimuli from corresponding brain responses. Generally, these methods map brain responses, captured in the form of functional magnetic resonance imaging (fMRI), to more common modalities suitable for feeding into pretrained vision-language models [22, 40, 55] for subsequent image reconstruction. For example, Lin et al. [28] project fMRI data to a CLIP [39] common space and reconstruct images through a finetuned StyleGAN2 [22]. Takagi and Nishimoto [44] utilize the ridge regression to link fMRI signals with CLIP text embeddings and the latent space of Stable Diffusion [40] (SD). Xia et al. [53] extract semantics, depth, and color cues, and reconstruct images using a depth-color-conditioned SD. Rather than relying solely on textual embeddings, several methods [16, 45] aim to obtain explicit descriptions for the visual stimuli. In contrast, our method decodes brain responses into various human-readable textual and visual cues, which can also flexibly serve as inputs for generative models.

**Multimodal Large Language Models.** Expanding Large Language Models (LLMs) to encompass other modalities, such as images, has garnered considerable attention recently. These models typically comprise three components: a frozen image encoder, a trainable adapter, and a frozen or finetuned LLM. The adapter's role is to bridge the gap from image features to the LLM, which can be implemented as a linear layer [9], a multilayer perceptron (MLP) [30], or a lightweight transformer [20]. In addition to vision-focused LLMs, recent studies aim to expand the boundaries of LLMs to include other modalities, making it possible to unify multiple modalities within a single LLM. Brain signals, as an emerging modality, have also recently been incorporated, for example in OneLLM [16], but like training with other modalities, all these methods require massive amounts of data and abundant computational resources. In contrast, we demonstrate a simple yet effective way to align brain signals with images.

**Multimodal-Brain Alignment.** The prevailing practice for brain alignment is to map the neural modality into a common latent space [16, 34, 41, 53], which can be divided into two lines of works: discriminative alignment and generative alignment. Considering the scarcity of data, methods in the first category aligns the brain modality within a pretrained embedding space, such as CLIP [39], through direct regression [34, 44, 45], contrastive learning [53], or diffusion prior [41]. The second is garnering significant attention in the field of MLLMs. For instance, OneLLM [16] adopts generative training to learn the alignment of multimodal inputs, including brain signals, thereby connecting a universal encoder with an LLM. However, such alignments between brain signals and images or text are trained per subject, resulting in one model for each subject. In contrast, we align the brain modality with image features to recover both semantic and spatial cues and achieve cross-subject multimodal-brain alignment to leverage user diversity.
<figure><img src="image_3.png" /><figcaption>**Fig. 2: Overview of UMBRAE.** Our brain encoder includes subject-specific tokenizers and a universal perceive encoder (Sec. 3.1). Neural signals (fMRI) from multiple subjects are mapped into a common feature space, enabling cross-subject training and weakly-supervised adaptation (Sec. 3.2). The brain encoder learns to align neural signals with image features (Sec. 3.3). During inference, the learned encoder interacts with MLLMs and performs brain understanding tasks according to given prompts (Sec. 3.4).</figcaption></figure>

# 3 UMBRAE

Our method is designed to address two shortcomings of the brain decoding literature. First, instead of learning a unimodal decoding (text or image), we learn a brain encoder that aligns brain features with pretrained multimodal space thus benefiting from all the MLLMs downstream tasks for multimodal decoding. Second, we observe that prior works train per-subject models owing to neuroscience showing inter-subject variability of the brain activities [2]. Instead, we intuit that inter-subject patterns present and learn a subject-unified representation by training jointly across-subjects. Beside better performance, this allows adaptation to novel subjects with minimal training data.

The overview of our method, named UMBRAE, is in Fig. 2. The acronym stands for unified multimodal brain decoding and signifies the process of unveiling encoded information hidden within the 'shadows' of brain signals. We rely mainly on a flexible brain encoder architecture (Sec. 3.1) and a cross-subject training strategy (Sec. 3.2) to map brain responses from different subjects, each with variable length, into a common feature space. In our experiments, we observe that simply binding brain modality with image features enables the recovery of both semantic and spatial cues (Sec. 3.3). We then conduct brain prompting interface by inputting the standardized brain representations from diverse subjects into MLLMs for downstream tasks (Sec. 3.4).

## 3.1 Architecture

Our brain encoder, based on a lightweight transformer architecture [20, 50], accommodates variable-length brain response inputs. This is important for cross-subject training as fMRI data are variable across subjects. Hence, our architecture
comprises subject-specific tokenizers which aim is to extract and map the subject-specific characteristics, along with a universal perceive encoder designed to capture subject-agnostic knowledge, later aligned with image features.

**Subject Tokenizer.** The subject-specific tokenizer projects the input brain signal $s \in \mathbb{R}^{1 \times L_s}$ selected from the subject set $S_\Omega$, with arbitrary length $L_s$, into a fixed-length sequence of brain tokens $\mathbf{x} \in \mathbb{R}^{L \times D}$. Here, $L$ is the sequence length and $D$ is the token dimension. Considering inter-subject variability in brain patterns [2], we design a separate tokenizer for each subject. Besides, we introduce a learnable subject-specific token ${s_k}_k^K$ to switch between subjects, where $K$ is the total number of subjects and $s_k \in \mathbb{R}^{M \times D}$ contains $M$ tokens of dimension $D$. Then, we prepend subject-specific tokens $s_k$ to the predicted brain tokens $\mathbf{x}$ and encode them with the following universal perceive encoder.

**Universal Perceive Encoder.** The universal perceive encoder seeks to project all brain tokens $\mathbf{x}$ into a common space. We utilize here a transformer-based architecture [20] which uses cross-attention modules to project the input tokens into a latent bottleneck where the key $K$ and value $V$ are projections of the input tokens, while $Q$ is the projection of learnable latent queries.

The subject-specific tokenizers are expected to capture specific information for each subject, including structural and functional differences among individual brains; and the universal perceive encoder aims to extract common knowledge across different subjects, encompassing categories, semantics, textures, and geometries of various objects and scenes. We now detail the training strategy for cross-subject alignment.

## 3.2 Cross-Subject Alignment

For cross-subject alignment, it is crucial to ensure that examples from each subject are uniformly sampled. This enables the model to avoid subject preference and prevent catastrophic forgetting. Therefore, we adopt a sampling strategy to ensure that $\theta$ percent of samples in a batch are from the same subject. Considering $S_\Omega$ being the union over $K$ subjects training data $S_\Omega = \cup_{k \in \{1, 2, \dots, K\}} S_k$, we select data samples from a subject $S_k$ with probability:

$$p_k = \frac{\|S_k\|}{\sum_{n=1}^{K} \|S_n\|}, \quad (1)$$

where $\| \cdot \|$ denotes cardinality (i.e., the number of data samples). To construct a batch with $B$ number of data samples, we select a subject $S_k$ with probability $p_k$ and conduct random sampling to yield $\theta \times B$ training samples. The remaining $(1-\theta) \times B$ examples are uniformly sampled from other subjects, i.e., $S_\Omega \setminus S_k$. This batch sampling strategy significantly benefits from user diversity as it allows the model to focus primarily on intra-subject training while being exposed to different subjects to improve inter-subject discrimination. We latter demonstrate that this cross-subject alignment enhances performance without incurring extra training costs compared to training with a single subject, at no additional training time.
## 3.3 Multimodal Alignment

Previous multimodal methods learn to map multiple modalities into a common latent space, typically through contrastive pretraining, using either images [15] or text [16] as the binding modality. Instead, we align brain representations with image features from a pretrained image encoder using element-wise reconstruction.

Given a brain response $s \in \mathbb{R}^{1 \times L_s}$ and the corresponding visual stimulus $v \in \mathbb{R}^{W \times H \times C}$, the source brain encoder $\mathcal{B}$ and target visual image encoder $\mathcal{V}$ encode brain signals and images into features denoted $\mathbf{b}$ and $\mathbf{v}$, respectively. We train the brain encoder $\mathcal{B}$ to minimize the distance between brain features and image features, aiming for a close approximation $\mathcal{B}(b) \approx \mathcal{V}(v)$ through:

$$ \mathcal{L}_{\text{rec}} = \mathbb{E}_{\mathbf{b} \sim \mathcal{B}, \mathbf{v} \sim \mathcal{V}} [\|\mathcal{V}(v) - \mathcal{B}(b)\|_2^2]. \quad (2) $$

Our brain encoder $\mathcal{B}$ learns the alignment between source brain space $\mathcal{B}$ and target image space $\mathcal{V}$. Different from previous methods, we align the brain signals with the intermediate image features from a pretrained image encoder, thus achieving semantic and spatial alignment for the brain representation. MLLMs [9, 30] show such features provide sufficient visual cues for finetuning LMMs. Furthermore, aligning with intermediate image features allows direct input of aligned brain representations into the MLLM.

## 3.4 Brain Prompting Interface

After alignment, brain features from the brain encoder $\mathcal{B}$ are fed into the MLLM's adapter, to retrieve the mapped visual embeddings `<image>`. These embeddings are then concatenated with a user instruction prompt and inputted into the finetuned LLM. Thus, our brain encoder inherits from multimodal capabilities of the MLLM, allowing tasks to be used in a prompting fashion using template:

system message. user: `<instruction>` `<image>` assistant: `<answer>`

The tags `<instruction>` and `<answer>` serve as placeholders for human instructions and assistant answers. We use variable templates for different tasks. Specifically, brain captioning uses 'Describe this image <image> as simply as possible.'; brain grounding, 'Locate <expr> in <image> and provide its coordinates, please.', where <expr> is the expression. More templates for different supported tasks can be found in the appendix.

Our primary focus in this study is on brain captioning and grounding, which reflects the capabilities of brain signals in concept recognition and spatial localization. They are often referred to as image captioning and visual grounding in the multimodal learning literature. However, in this context, brain signals are used as the input rather than images. Our method also supports other instruction-following capabilities, such as conversation, detail description, and complex reasoning. Our method is model-agnostic, allowing for the use of any image encoders, LLMs, and MLLMs according to specific needs.
# 4 Experiments

## 4.1 Implementation Details

**Architecture.** We use the pretrained CLIP ViT-L/14 [39] as the visual encoder and Vicuna-7B/13B [11] as the LLM, consistent with the setup in Shikra [9] and LLaVA [30]. The target image features are obtained from the second last layer of the transformer encoder, denoted as $T \in \mathbb{R}^{16 \times 16 \times 1024}$, which are then converted to $T' \in \mathbb{R}^{256 \times D}$ for further processing by the adapter and LLM. The dimension $D$ is 4,096 for Vicuna-7B and 5,120 for Vicuna-13B. The learnable tokens for each subject are of dimensions $\mathbb{R}^{5 \times 1024}$.

**Training Details.** Our models are trained on a single A100 GPU for 240 epochs with a global batch size of 256. It takes around 12 hours to complete. We use AdamW [31] as the optimizer with $\beta_1 = 0.9$, $\beta_2 = 0.95$, and weight decay of 0.01. For the learning rate scheduler, we use one-cycle [42] with an initial learning rate of 3e-4. We set $\theta = 0.5$, meaning that in each batch of 256 samples, 128 come from each of two subjects. The selection probabilities are identical for each subject, as they contain the same number of training data. Following visual decoding studies [41, 44, 53], we use the standard train and test splits for the four subjects (S1, S2, S5, S7). Specifically, each subject contains 24,980 training samples. For testing, we report the average of the three same-image repetitions, totaling 982 samples per subject. Note that the above studies train a subject-specific model for each of the four subjects, while we train one brain encoder for them all.

## 4.2 BrainHub

For evaluation, we construct a multimodal brain understanding benchmark, BrainHub, to further analyze the information contained in brain signals. Specifically, we extend the NSD [2], a popular dataset comprising brain responses of subjects viewing visual stimuli (images) sourced from Microsoft Common Objects in Context (COCO) [29]. NSD provides (fMRI, image) pairs which is sufficient for visual decoding. However, we aim to explore the ability to process brain signals for identifying visual concepts, recognizing and localizing instances, as well as extracting spatial relationships among multiple exemplars. Specifically, we process the corresponding COCO images for each fMRI sample and extract relevant labels for the following tasks and metrics:

- **Brain Captioning** aims at textually describing the primary content of a given brain response. Ground truth captions are retrieved from COCO [29], and evaluation of inferred captions uses five standard metrics: BLEU-$k$ [35], METEOR [5], ROUGE-L [27], CIDEr [51], and SPICE [3], in addition to two CLIP-based scores [39], namely CLIP-S and RefCLIP-S [17].

- **Brain Grounding** is the counterpart of visual grounding [9, 30] and seeks to recover spatial locations from brain signals by inferring coordinates. Given that identified classes might be named differently, or simply absent from
<table><caption>**Table 1: Brain Captioning.** 'UMBRAE-S1' refers to our model trained with a single subject (S1 here) only, while 'UMBRAE' denotes the model with cross-subject training. 'Shikra-w/img' refers to the image captioning result from Shikra [9] using the ground truth image as input, serving as an approximate upper bound. The colors represent the best, second-best, and third-best performance. Brain captioning results for the other subjects are provided in the appendix.</caption><thead><tr><th>Method</th><th>BLEU1</th><th>BLEU2</th><th>BLEU3</th><th>BLEU4</th><th>METEOR</th><th>ROUGE</th><th>CIDEr</th><th>SPICE</th><th>CLIP-S</th><th>RefCLIP-S</th></tr></thead><tbody><tr><td>Shikra-w/img [9]</td><td>82.38</td><td>69.90</td><td>58.63</td><td>49.66</td><td>35.60</td><td>65.49</td><td>161.43</td><td>27.62</td><td>80.60</td><td>85.92</td></tr><tr><td>SDRecon [44]</td><td>36.21</td><td>17.11</td><td>7.72</td><td>3.43</td><td>10.03</td><td>25.13</td><td>13.83</td><td>5.02</td><td>61.07</td><td>66.36</td></tr><tr><td>OneLLM [16]</td><td>47.04</td><td>26.97</td><td>15.49</td><td>9.51</td><td>13.55</td><td>35.05</td><td>22.99</td><td>6.26</td><td>54.80</td><td>61.28</td></tr><tr><td>UniBrain [32]</td><td>-</td><td>-</td><td>-</td><td>-</td><td>16.90</td><td>22.20</td><td>-</td><td>-</td><td>-</td><td>-</td></tr><tr><td>BrainCap [14]</td><td>55.96</td><td>36.21</td><td>22.70</td><td>14.51</td><td>16.68</td><td>40.69</td><td>41.30</td><td>9.06</td><td>64.31</td><td>69.90</td></tr><tr><td>UMBRAE-S1</td><td>57.63</td><td>38.02</td><td>25.00</td><td>16.76</td><td>18.41</td><td>42.15</td><td>51.93</td><td>11.83</td><td>66.44</td><td>72.12</td></tr><tr><td>UMBRAE</td><td>59.44</td><td>40.48</td><td>27.66</td><td>19.03</td><td>19.45</td><td>43.71</td><td>61.06</td><td>12.79</td><td>67.78</td><td>73.54</td></tr></tbody></table>

ground truth labels, we evaluate bounding boxes through the task of referring expression comprehension [56], using accuracy and intersection over union (IoU) as the evaluation metrics.

- **Brain Retrieval** is to search for pertinent results in response to a provided query from a large database, often considered as a form of fine-grained, instance-level classification. The evaluation metric used is accuracy.

- **Visual Decoding** refers to the capability to reconstruct the visual stimuli associated with the fMRI data. We include it here for consistency with the extensive literature on visual decoding [34, 41].

## 4.3 Brain Captioning

Tab. 1 provides an evaluation of our brain captioning for subject 1 (S1), with respect to SOTA baselines being SDRecon [45], BrainCap [14] and OneLLM [16]. From the latter table, UMBRAE outperforms all baselines by a significant margin on all metrics. SDRecon poor performance results from its limited limited vocabulary, and the use redundant or meaningless words in its captioning, such as 'person and person with person person wearing a tie shirt person person, women's clothing.' , which impacts the quality metrics negatively. BrainCap [14] follows a similar pipeline but replaces the captioning model, which performs better. OneLLM [16] learns a unified encoder for multimodal-text alignment which improves the caption quality but deteriorates the CLIP similarity score, as it merely aligns with texts. In contrast, the alignment with image features of UMBRAE preserves more accurate semantic and spatial cues decoded from the brain signals. Moreover, the use of LLMs helps generate sentences that are fluent, complete, and rich in information. Interestingly, we note that the performance of UMBRAE (trained on S1, S2, S5, S7) exceeds those when trained only on data from S1 (UMBRAE-S1), demonstrating the ability to learn from cross-subject patterns. As an approximate upper bound, we also report 'Shikra-w/img' which, similar to us, utilizes Shikra [9] for captioning though here using the ground truth image (visual stimuli). Results for other subjects are provided in the appendix.
<figure><img src="image_4.png" /><figcaption>**Fig. 3: Example Results.** Our method inherits the multimodal capability from MLLMs and thus supports multiple brain captioning and grounding tasks. Different task prompts for the same input brain signal result in unique outcomes.</figcaption></figure>

## 4.4 Brain Grounding

Our method showcases grounding capabilities across various settings by adapting to the corresponding instructions, which is illustrated in Fig. 3. For instance, we conduct spotting captioning, a task aimed at generating a description of the image along with bounding boxes for the mentioned items, using the instruction ‘Please interpret this image and provide coordinates [x1,y1,x2,y2] for each object you mention’. We can also perform referring expression comprehension using ‘could you find and tell me the coordinates of <expr>?’. For evaluation, we detect queried objects and report the accuracy and IoU. The accuracy metric ‘acc@m’ measures the percentage of correctly labeled instances with an IoU greater than the threshold *m*. Results of acc@0.5 are reported in Tab. 2 and examples are depicted in Fig. 3. More results are in the appendix.

In Tab. 2 we again report an approximate upper bound ‘Shikra-w/img’ being the visual grounding using the ground truth image. Given the absence of prior brain grounding baselines, we construct natural baselines by combining Shikra with the images from SOTA visual decoding methods [34, 41, 53], referred as ‘Shikra-w/method’. We also report ‘Shikra-w/UMBRAE’ using our own visual decoding later described in Sec. 4.6. Being the first to attempt decoding spatial information from brain signals, our method ‘UMBRAE’ performs roughly on par with our constructed baselines while being at least x10 faster for grounding noting that speed is a critical characteristic for brain controlled applications.

In addition to metrics for all classes denoted ‘All’, we inspire from neuroscience exploring the salience-processing systems in the human brain [49] for more detailed evaluation. Specifically, we group the 80 classes of COCO [29] according to their prominence into: ‘Salient’, being the union of ‘Salient Creatures’ (people and animals) and ‘Salient Objects’ (e.g., car, bed, table), and ‘Inconspicuous’ (e.g., backpack, knife, toothbrush). We report the detailed mapping in the appendix. It is interesting to note that UMBRAE is outperformed on Salient Creatures but not on Salient Objects and Inconspicuous elements. This suggests that visual
<table><caption>**Table 2: Brain Grounding.** 'UMBRAE-Sx' refers to our model trained with a single subject only, while 'UMBRAE' denotes the model with cross-subject training. 'Shikra-w/img' refers to the visual grounding result from Shikra [9] using the ground truth visual stimuli (images) as input. Similarly, 'Shikra-w/method' provides visual grounding results using images produced by visual decoding methods [34, 41, 53]. We highlight best, second-best, and third-best performance per subject.</caption><thead><tr><th>Method</th><th>Eval</th><th>All acc@0.5</th><th>IoU</th><th>Salient acc@0.5</th><th>IoU</th><th>Salient acc@0.5</th><th>Creatures</th><th>IoU</th><th>Salient acc@0.5</th><th>Objects</th><th>IoU</th><th>Inconspicuous acc@0.5</th><th>IoU</th><th>Time (s)</th></tr></thead><tbody><tr><td>Shikra-w/img [9]</td><td>*</td><td>51.96</td><td>47.22</td><td>62.92</td><td>56.44</td><td>66.71</td><td>59.34</td><td>58.79</td><td>53.27</td><td>38.29</td><td>35.71</td><td>0.96</td><td></td><td></td></tr><tr><td>Shikra-w/BrainDiffuser [34]</td><td></td><td>17.49</td><td>19.34</td><td>27.18</td><td>27.46</td><td>38.71</td><td>34.63</td><td>14.62</td><td>19.66</td><td>5.39</td><td>9.20</td><td>16.4</td><td></td><td></td></tr><tr><td>Shikra-w/MindEye [41]</td><td></td><td>15.34</td><td>18.65</td><td>23.83</td><td>26.96</td><td>29.29</td><td>31.64</td><td>17.88</td><td>21.86</td><td>4.74</td><td>8.28</td><td>16.4</td><td></td><td></td></tr><tr><td>Shikra-w/DREAM [53]</td><td>S1</td><td>16.21</td><td>18.65</td><td>26.51</td><td>27.35</td><td>34.43</td><td>33.85</td><td>17.88</td><td>20.28</td><td>3.35</td><td>7.78</td><td>10.5</td><td></td><td></td></tr><tr><td>Shikra-w/UMBRAE</td><td></td><td>16.83</td><td>18.69</td><td>27.10</td><td>27.55</td><td>34.14</td><td>33.65</td><td>19.44</td><td>20.92</td><td>4.00</td><td>7.64</td><td>16.4</td><td></td><td></td></tr><tr><td>UMBRAE-S1</td><td></td><td>13.72</td><td>17.56</td><td>21.52</td><td>25.14</td><td>26.00</td><td>29.06</td><td>16.64</td><td>20.88</td><td>4.00</td><td>8.08</td><td>0.92</td><td></td><td></td></tr><tr><td>UMBRAE</td><td></td><td>18.93</td><td>21.28</td><td>30.23</td><td>30.18</td><td>39.57</td><td>36.64</td><td>20.06</td><td>23.14</td><td>4.83</td><td>10.18</td><td>0.92</td><td></td><td></td></tr><tr><td>UMBRAE-S2</td><td>S2</td><td>15.21</td><td>18.68</td><td>23.60</td><td>26.59</td><td>27.86</td><td>30.51</td><td>18.97</td><td>22.32</td><td>4.74</td><td>8.81</td><td>-</td><td></td><td></td></tr><tr><td>UMBRAE</td><td></td><td>18.27</td><td>20.77</td><td>28.22</td><td>29.19</td><td>38.29</td><td>36.13</td><td>17.26</td><td>21.63</td><td>5.86</td><td>10.25</td><td>-</td><td></td><td></td></tr><tr><td>UMBRAE-S5</td><td>S5</td><td>14.72</td><td>18.45</td><td>22.93</td><td>26.34</td><td>26.86</td><td>29.84</td><td>18.66</td><td>22.52</td><td>4.46</td><td>8.60</td><td>-</td><td></td><td></td></tr><tr><td>UMBRAE</td><td></td><td>18.19</td><td>20.85</td><td>28.74</td><td>30.02</td><td>36.71</td><td>36.25</td><td>20.06</td><td>23.23</td><td>5.02</td><td>9.41</td><td>-</td><td></td><td></td></tr><tr><td>UMBRAE-S7</td><td>S7</td><td>13.60</td><td>17.83</td><td>21.07</td><td>25.19</td><td>24.57</td><td>28.90</td><td>17.26</td><td>21.15</td><td>4.28</td><td>8.64</td><td>-</td><td></td><td></td></tr><tr><td>UMBRAE</td><td></td><td>16.74</td><td>19.63</td><td>25.69</td><td>27.90</td><td>33.14</td><td>33.42</td><td>17.57</td><td>21.89</td><td>5.58</td><td>9.31</td><td>-</td><td></td><td></td></tr></tbody></table>

decoding effectively reconstruct the salient creatures in the image space, arguably because the subject focuses on the latter.

Experimentally, we also notice that images containing few salient objects exhibit better performance compared to cluttered scenes, and easy background also lead to better grounding. Conversely, we note that localization suffers when images are filled with numerous inconspicuous objects. We argue that inconspicuous objects in the image may not draw the subject's attention, or that relevant brain activities may not be effectively captured during experiments [2]. Our categorization and observation also align with the semantic selectivity found in the higher visual cortex of the human brain [13, 21, 38], which contains specialization of certain regions that respond selectively to specific semantic categories of visual stimuli, such as faces, bodies, words, food, and places. The results demonstrate that our method performs well in relevant cases.

## 4.5 Brain Retrieval

The retrieval evaluation demonstrates the amount of image-specific information contained in the brain embedding. Following [41], we conduct three experiments: forward retrieval, backward retrieval, and exemplar retrieval. The *forward* retrieval computes accuracy of identifying the correct paired CLIP image embedding from 300 brain embeddings. Conversely, the *backward* retrieval finding the correct brain embedding from 300 image embeddings. For a fair comparison, we modify the output dimension and proceed to optimize the encoder and embedding using an InfoNCE [33] loss. We follow the same procedure as in [28] for calculating the retrieval metrics reported in Tab. 3. The *exemplar* retrieval aims to find the exact original image within the 982 test images. Our method outperforms current methods with accuracy percentages of 94.2%, 91.3%, and 93.8% on

<footnote>* The subjects test sets use the same reference images making 'Shikra-w/img' identical for all subjects.</footnote>
<table><caption>**Table 3: Brain Retrieval.** We report *forward*, *backward*, and *exemplar* retrieval metrics [41], showing that our method significantly outperforms the baselines. We also compare the floating-point operations (FLOPs), multiply-accumulate operations (MACs), and model parameters (Params). ‘UMBRAE’ denotes the model with cross-subject training. Colors represent the best and second-best performance.</caption><thead><tr><th>Method</th><th>Forward</th><th>Backward</th><th>Exemplar</th><th>FLOPs (G)</th><th>MACs (G)</th><th>Params (M)</th></tr></thead><tbody><tr><td>MindReader [28]</td><td>11.0%</td><td>49.0%</td><td>\</td><td>\</td><td>\</td><td>\</td></tr><tr><td>BrainDiffuser [34]</td><td>21.1%</td><td>30.3%</td><td>\</td><td>\</td><td>\</td><td>\</td></tr><tr><td>MindEye [41]</td><td>93.6%</td><td>90.1%</td><td>93.2%</td><td>52.27</td><td>26.13</td><td>1,003.64</td></tr><tr><td>UMBRAE</td><td>94.2%</td><td>91.3%</td><td>93.8%</td><td>67.48</td><td>33.72</td><td>146.24</td></tr></tbody></table>

<table><caption>**Table 4: Visual Decoding.** Following the standard evaluation metrics [34], our method performs on par or better than the SOTA methods [28, 34, 41, 44, 53]. Colors represent the best, second-best, and third-best performance.</caption><thead><tr><th rowspan="2">Method</th><th colspan="4">Low-Level</th><th colspan="4">High-Level</th></tr><tr><th>PixCorr ↑</th><th>SSIM ↑</th><th>AlexNet(2) ↑</th><th>AlexNet(5) ↑</th><th>Inception ↑</th><th>CLIP ↑</th><th>EffNet-B ↓</th><th>SwAV ↓</th></tr></thead><tbody><tr><td>MindReader [28]</td><td>-</td><td>-</td><td>-</td><td>-</td><td>78.2%</td><td>-</td><td>-</td><td>-</td></tr><tr><td>SDRecon [44]</td><td>-</td><td>-</td><td>83.0%</td><td>83.0%</td><td>76.0%</td><td>77.0%</td><td>-</td><td>-</td></tr><tr><td>BrainDiffuser [34]</td><td>.254</td><td>.356</td><td>94.2%</td><td>96.2%</td><td>87.2%</td><td>91.5%</td><td>.775</td><td>.423</td></tr><tr><td>MindEye [41]</td><td>.309</td><td>.323</td><td>94.7%</td><td>97.8%</td><td>93.8%</td><td>94.1%</td><td>.645</td><td>.367</td></tr><tr><td>DREAM [53]</td><td>.274</td><td>.328</td><td>93.9%</td><td>96.7%</td><td>93.4%</td><td>94.1%</td><td>.645</td><td>.418</td></tr><tr><td>UMBRAE</td><td>.283</td><td>.341</td><td>95.5%</td><td>97.0%</td><td>91.7%</td><td>93.5%</td><td>.700</td><td>.393</td></tr></tbody></table>

forward, backward, and exemplar retrieval experiments, respectively. These results demonstrate the ability to distinguish among misconstruable exemplars and suggest the fine-grained, image-specific information retained in the predicted brain embeddings.

## 4.6 Visual Decoding

Although this is not our primary purpose, to show the versatile capabilities of our method, we conduct experiments on the visual decoding task and compare with SOTAs [28, 34, 41, 44] using recognized metrics. While our method is not specifically tailored for this task, the textual and visual outputs it generates can be used as cues for fMRI-to-image reconstruction. Our results in Tab. 4 is utilizing the Versatile Diffusion [55] to reconstruct the image based on the decoded text and CLIP image embedding obtained in Sec. 4.5. We employ the same evaluation metrics as used in [34]. Specifically, PixCorr calculates the pixel-level correlation between the ground-truth and reconstructed images. SSIM [52] measures the textural and structural similarity instead of pixel-wise differences. Two-way comparisons are conducted using the second and fifth layers of AlexNet [23], the last pooling layer of Inceptionv3 [43], and the last layer of CLIP ViT-L/14 image encoder [39]. EffNet-B and SwAV are distances from EfficientNet [46] and SwAV-ResNet50 [6]. The first four metrics focus on low-level characteristics, whereas the remaining metrics are concerned with higher-level measurements.

Results in Tab. 4 demonstrate that our method performs comparably or better than state-of-the-art methods without any specific designs tailored for this task.
<figure><img src="image_5.png" /><figcaption>**Fig. 4: Weakly-Supervised Subject Adaptation.** This model for S7 is trained or finetuned on a pretrained model (trained on S1, S2, and S5) using varying ratios (0.05, 0.1, 0.2, 0.3, 0.5, 0.8, 1.0) of training data. The model achieves comparable performance using only 30% of the data and obtains better results when increasing the ratio of used training samples to 50%, compared to the model trained on the full dataset of S7.</figcaption></figure>

Moreover, with access to common conditions like texts, image embeddings, and bounding boxes, we can leverage a wide range of pretrained image generation models. These models encompass text-to-image (e.g., SD [40], SD-XL [37]), layout-to-image [26], and multiple-condition [55]. Details are in the appendix.

## 4.7 Weakly-Supervised Adaptation

Capturing brain signals, such as high-resolution fMRI, requires specialized equipment and professional personnel, making it challenging to collect on a large scale. A benefit of our cross-subjects training is to allow subject adaptation with minimal training data. To evaluate this emerging property, we train our brain encoder with subjects S1, S2, and S5 and seek to adapt the trained to a new subject S7 using various amount of training data. For ablation, we explore two settings where we train a new tokenizer for S7 with the universal perceve encoder being either *Frozen* or *Finetuned*.

Plots in Fig. 4 report ‘Frozen’ and ‘Finetuned’ adaptation with variable amount of S7 data. Additionally, we report ‘UMBRAE’ when trained with all training data of {S1, S2, S5, S7} as well as ‘UMBRAE-S7’ trained on all S7 data only. Notably, compared to ‘UMBRAE-S7’, our ‘Finetuned’ adaptation achieves comparable performance using only 30% of the data and often better when using more than 50%. Training only the tokenizer while keeping the pretrained backbone encoder frozen generally resulted in lower performance compared to fine-tuning the backbone together. This could be because the backbone encoder did not adequately incorporate the subject discrepancy in S7. Please consult the appendix for extra results and discussions on other subjects.
<table><caption>**Table 5: Ablation Study.** 'MLP' refers to the MLP-based brain encoder [41], while 'Enc-S' and 'Enc-U' represent our transformer-based encoders for single and multiple subjects, respectively. 'Dim.' means the output dimension of the brain encoder. The output needs to be passed to the adapter for further processing if it is 1024; otherwise, it is directly inputted into the LLM. The adapter has three training settings: 'Pretrained' means freezing the weights, 'Finetuned' means finetuning based on the pretrained weights, and 'Joint' means training with the encoder from scratch. 'Loss Type' refers to loss functions (MSE, NCE, or both) applied to the outputs from the encoder (E.) or the adapter (A.).</caption><thead><tr><th rowspan="2">Arch.</th><th rowspan="2">Dim.</th><th rowspan="2">Adapter</th><th rowspan="2">Loss Type</th><th colspan="5">Captioning</th><th colspan="4">Grounding</th></tr><tr><th>BLEU1</th><th>CIDEr</th><th>SPICE</th><th>CLIP-S</th><th>RefCLIP-S</th><th>All acc@0.5</th><th>IoU</th><th>Salient acc@0.5</th><th>IoU</th></tr></thead><tbody><tr><td>MLP</td><td>1024</td><td>Pretrained</td><td>MSE (E.)</td><td>55.04</td><td>46.24</td><td>10.80</td><td>64.75</td><td>70.59</td><td>13.44</td><td>17.54</td><td>20.55</td><td>24.68</td></tr><tr><td>MLP</td><td>1024</td><td>Finetuned</td><td>MSE (A.)</td><td>54.02</td><td>43.24</td><td>10.35</td><td>64.09</td><td>70.02</td><td>13.56</td><td>17.91</td><td>20.92</td><td>25.54</td></tr><tr><td>Enc-S</td><td>1024</td><td>Pretrained</td><td>MSE (E.)</td><td>57.63</td><td>51.93</td><td>11.83</td><td>66.44</td><td>72.12</td><td>13.72</td><td>17.56</td><td>21.52</td><td>25.14</td></tr><tr><td>Enc-S</td><td>4096</td><td>N/A</td><td>MSE (A.)</td><td>52.06</td><td>36.40</td><td>9.06</td><td>62.30</td><td>68.27</td><td>13.31</td><td>17.04</td><td>20.85</td><td>24.78</td></tr><tr><td>Enc-S</td><td>1024</td><td>Joint</td><td>MSE (A.)</td><td>55.02</td><td>43.53</td><td>10.48</td><td>64.00</td><td>70.01</td><td>13.72</td><td>17.57</td><td>21.44</td><td>25.15</td></tr><tr><td>Enc-S</td><td>1024</td><td>Joint</td><td>MSE (E.) NCE (A.)</td><td>27.09</td><td>3.16</td><td>1.27</td><td>52.69</td><td>59.08</td><td>8.72</td><td>11.40</td><td>13.78</td><td>16.26</td></tr><tr><td>Enc-S</td><td>1024</td><td>Joint</td><td>MSE (A.) NCE (A.)</td><td>51.69</td><td>34.09</td><td>8.71</td><td>62.27</td><td>68.05</td><td>13.68</td><td>18.07</td><td>21.07</td><td>25.45</td></tr><tr><td>Enc-U</td><td>1024</td><td>Pretrained</td><td>MSE (E.)</td><td>59.44</td><td>61.06</td><td>12.79</td><td>67.78</td><td>73.54</td><td>18.93</td><td>21.28</td><td>30.23</td><td>30.18</td></tr></tbody></table>

# 5 Ablation Study

## 5.1 Architectural Improvements

**MLP-based Encoder vs. Our Encoder.** The MLP-based brain encoder is adapted from [41] with slight adjustments to match the desired output dimension. This deep MLP backbone amounts to 1,003.64 million parameters per subject. In comparison, our model needs only 112.63 million parameters for a single subject and 146.24 million for all four subjects. This translates to an 88.78% reduction in parameters for a single subject and a 96.36% reduction for all four subjects, respectively. Our single-subject encoder (denoted as UMBRAE-Sx) surpasses the MLP-based architecture [41] in captioning (Tab. 1), grounding (Tab. 2), and retrieval (Tab. 3) tasks by significant margins. The universal encoder (denoted as UMBRAE) achieves even greater performance improvements.

**Single vs. Cross-Subject Design.** The universal encoder differs from the single-subject encoder solely in the addition of subject-specific tokenizers (Sec. 3.1), and its training only varies in the batch sampling strategy (Sec. 3.2), enabling the training of diverse subjects within one model. Additionally, the resources required are basically the same as those for training a single-subject model, eliminating the necessity of extra training time or computational resources. This cross-subject design benefits from user diversity, achieving superior performance compared to focusing on a single subject. Results in Tabs. 1 to 3 show that the cross-subject design surpasses its single-subject counterpart across almost metrics.

## 5.2 Training Strategies

Current vision-language models typically comprise three main components: an image encoder, an adapter, and a large language model. Within this framework, there are several potential ways for multimodal-brain alignment. For example,
one could train the model to map brain responses to a pretrained semantic space through contrastive learning, which has been a common practice in previous methods. Alternatively, one could opt to fine-tune the adapter or train it jointly with the brain encoder, applying separate losses to each component. In this section, we delve into the motivation and rationale behind aligning with image features from the image encoder using a reconstruction loss. Further, in the appendix we ablate our sampling strategy.

**Pretrained vs. Finetuned Adapter.** The adapter serves as the bridge connecting multimodal encoders [16,39] with the output space of MLLMs [9,30,48]. As shown in Tab. 5, either finetuning the adapter or training it jointly with the brain encoder results in decreased performance, likely due to catastrophic forgetting that occurs when updating well-trained parameters that have been learned from a significantly larger volume of data.

**Image Feature vs. LLM Embedding.** In addition to aligning with image features, we also conduct experiments on learning to align brain responses with the pretrained LLM used. Specifically, we explore three variants: (a) training the brain encoder and finetuning the adapter with different losses, (b) training the brain encoder and the adapter jointly, and (c) adjusting the output dimension of the brain encoder to align directly with the language embedding. As illustrated in Tab. 5, all attempts yield less desirable results compared to simply aligning with image features. Finetuning the entire model with new and sufficient data may indeed achieve better results, but acquiring such data can be challenging. However, experiments suggest that aligning with image features yields the best results when only image-brain pairs are available for training.

**Reconstruction vs. Contrastive Learning.** We further explore the effects of different loss functions on the aforementioned three variants, specifically a pixel-wise reconstruction loss (MSE) and a contrastive loss with MixCo augmentation (NCE) [33]. We also test applying separate losses to outcomes from the encoder and the adapter when trained jointly. The findings show that applying an MSE loss to the image feature while keeping the adapter unchanged leads to the most favorable performance in both concept recognition and object localization. Conversely, employing contrastive learning significantly diminishes performance.

# 6 Conclusion

In this work, we propose a method that decodes multimodal explanations from brain signals. Specifically, we introduce a universal brain encoder for multimodal brain alignment, which enables the recovery of conceptual and spatial details using multimodal large language models. To overcome unique brain patterns among different individuals, we introduce a novel cross-subject training strategy. This enables brain signals from multiple subjects to be trained within the same model and allows weakly-supervised subject adaptation, facilitating the training of a model for a new subject in a data-efficient manner. For evaluation, we construct BrainHub, a brain understanding benchmark, to facilitate future research.

**Acknowledgements.** This work was supported by the Engineering and Physical Sciences Research Council [grant EP/W523835/1] and the UKRI Future Leaders Fellowship [grant G104084].
# Appendix

In this appendix, we provide further insights, experiments and analyses. First, in Appendix A we describe the tasks and associated prompts used in UMBRAE and in Appendix B describe the BrainHub benchmark construction. We then report in Appendix C additional experiments to showcase the superiority of UMBRAE on all tasks, and extend our ablation and analyses in Appendix D. Finally, in Appendix E, we discuss the method limitations and potential negative impacts.

## A UMBRAE: Tasks and Example Prompts

Our method inherits multimodal understanding capabilities of MLLMs, enabling the switch between different tasks through the use of various task prompts. Taking Shikra [9] as an example, we excerpts the task prompts used during their training process in Tab. 1. Three prompts for each task are randomly selected to provide readers an intuitive understanding. During inference, users are not constrained to these predefined formats and are free to frame their queries in natural language, allowing for a broad spectrum of diverse and compelling task formats.

It should be noted that our method is compatible with nearly all tasks featured in Shikra [9] and LLaVA [30], with the exception of tasks that necessitate initially locating an object or scene as inputs, such as grounding captioning and referring expression generation. Importantly, note that although the term ‘image’ is used in the prompts we do not utilize images as input in our brain understanding tasks. The reference images are only used for visualization purposes. In practice, the prompt embedding will be concatenated with the brain features predicted from our brain encoder, utilizing brain signals as inputs. We report detailed results for each task of Tab. 1 later in Appendix C.

<table><caption>**Table 1: Supported Task Prompts.** The tags `<image>`, `<question>`, and `<expr>` are placeholders, representing input images, questions in QA tasks, and expressions in the REC task. During inference, users are free to create diverse task formats according to actual needs. Q, A, C, and $C^{\text{Box}}$ denote the **Question**, **Answer**, **Chain of thoughts** (CoT), and **CoT with Box**. CoT is delivering an answer along with the reasoning process. ‘Box’ denotes coordinates of bounding boxes.</caption><thead><tr><th>Task</th><th>Three randomly chosen examples from hundreds.</th></tr></thead><tbody><tr><td>Captioning</td><td>Describe this image &lt;image&gt; as simply as possible.<br>What is the content of the image &lt;image&gt;? Please answer in short sentences.<br>Summarize the content of the photo &lt;image&gt;</td></tr><tr><td>REC</td><td>In the given &lt;image&gt;, could you find and tell me the coordinates of &lt;expr&gt;?<br>I need the coordinates of &lt;expr&gt; in &lt;image&gt;, can you please assist me with that?<br>Locate &lt;expr&gt; in &lt;image&gt; and provide its coordinates, please.</td></tr><tr><td>Spotting Cap.</td><td>Can you provide a description of the image &lt;image&gt; and include the coordinates [x0,y0,x1,y1] for each mentioned object?<br>Please explain what's happening in the photo &lt;image&gt; and give coordinates [xmin,xmin,xmax,xmax] for the items you reference.<br>How would you describe the contents of the image &lt;image&gt;? Please provide the positions of mentioned objects in square brackets.</td></tr><tr><td>Q → A</td><td>I want to know the answer to: &lt;question&gt; Refer to the image &lt;image&gt; and give a clear response.<br>Answer this question directly after referring to the image &lt;image&gt;: &lt;question&gt;<br>Examine the image &lt;image&gt; and provide a brief answer for: &lt;question&gt;</td></tr><tr><td>Q → CA</td><td>Having a look at image &lt;image&gt;, can you tell me the answer to my question: &lt;question&gt; and the logic leading to it?<br>Please answer the following question: &lt;question&gt; based on the image &lt;image&gt;, and describe your thought process.<br>Upon analyzing the image &lt;image&gt;, please find the answer to my question: &lt;question&gt; and provide a detailed explanation.</td></tr><tr><td>Q → $C^{\text{Box}}$A</td><td>&lt;question&gt; Please offer your reasoning process, and provide bounding boxes of mentioned objects within square brackets. Here is the picture &lt;image&gt;<br>Please explain your reasoning and provide bounding boxes, denoted by square brackets, for the objects mentioned in the picture &lt;image&gt;: &lt;question&gt;<br>Consider &lt;image&gt;, and then provide a well-reasoned answer to: &lt;question&gt; Don't forget to mark relevant object locations using [x0,y0,x1,y1].</td></tr></tbody></table>
# B Details on BrainHub

Our multimodal brain understanding benchmark, coined BrainHub, extends the popular NSD [2] using COCO [29] annotations. Here, we first describe NSD (Appendix B.1) and then elaborate the construction of BrainHub (Appendix B.2).

## B.1 Natural Scenes Dataset

Natural Scenes Dataset [2] (NSD) stands as the largest publicly accessible fMRI dataset. It encompasses brain activity recordings from 8 subjects (participants) who viewed images passively for up to 40 hours inside an MRI machine. Each image was displayed for three seconds and repeated three times across 30-40 scanning sessions, yielding 22,000-30,000 fMRI response trials per subject.

Following recent studies, we utilize the four subjects who finished all scanning sessions, that is: S1, S2, S5, and S7. As in [41, 45, 53], we utilize preprocessed voxels corresponding to the 'nsdgeneral' brain region. The latter region, described by the NSD authors, comprises the subset of voxels in the posterior cortex most responsive to the presented visual stimuli. In the training set for each subject, there are 8,859 images and 24,980 fMRI trials (with each image tested up to three times). We compute the average response as per previous studies [41]. The test set comprises an additional 982 images and 2,770 fMRI trials common across four individuals. Importantly, all images used during the fMRI recordings are from the COCO [29] dataset, which we conveniently use to retrieve the original COCO labels to construct our BrainHub benchmark.

<table><caption>**Table 2: BrainHub Details.** The test set characteristics include the number of images, voxels, captions, bounding boxes, regions of interest (ROIs) in the fMRI data, subject references, and their corresponding dimensions.</caption><thead><tr><th>Images</th><th>Classes</th><th>Captions</th><th>Bounding boxes</th><th>ROIs</th><th>Subject</th><th>Dimension</th></tr></thead><tbody><tr><td rowspan="4">982</td><td rowspan="4">80</td><td rowspan="4">4,913</td><td rowspan="4">5,829</td><td>V1, V2, V3, hV4,</td><td>S1</td><td>15,724</td></tr><tr><td>VO, PHC, MT,</td><td>S2</td><td>14,278</td></tr><tr><td>MST, LO, IPS</td><td>S5</td><td>13,039</td></tr><tr><td></td><td>S7</td><td>12,682</td></tr></tbody></table>

## B.2 Benchmark Construction

BrainHub extends NSD [2] for the four subjects who finished all scanning sessions (S1, S2, S5, and S7). Tab. 2 outlines the characteristics of the benchmark. Each image viewed by the subject contains several captions and may include multiple bounding boxes (instances) of each class. According to their salience in images [49], we group the 80 classes of COCO into 4 salience categories: Salient, Salient Creatures, Salient Objects, and Inconspicuous. Note that 'Salient' is the combination of Salient Creatures and Objects. Fig. 1 shows the statistics and mapping of our categories, w.r.t. COCO classes. The inconspicuous (I) category accounts for the largest proportion, while the salient objects (SO) and creatures
(SC) are roughly even. The classes within each category exhibit significant imbalance; for instance, the number of instances for the 'person' class stands out in the salient category.

<figure><img src="image_6.png" /></figure>

<table><caption>(b) Mapping between categories and COCO classes.</caption><thead><tr><th>Category</th><th>COCO classes (# of classes)</th></tr></thead><tbody><tr><td>A</td><td>S + I (80)</td></tr><tr><td>S</td><td>SC + SO (28)</td></tr><tr><td>SC</td><td>person, bird, cat, dog, horse, sheep, cow, elephant, bear, zebra, giraffe (11)</td></tr><tr><td>SO</td><td>bicycle, car, motorcycle, airplane, bus, train, truck, boat, bench, chair, couch, bed, dining table, toilet, sink, refrigerator, clock (17)</td></tr><tr><td>I</td><td>traffic light, fire hydrant, stop sign, parking meter, backpack, umbrella, handbag, tie, suitcase, frisbee, skis, snowboard, sports ball, kite, baseball bat, baseball glove, skateboard, surfboard, tennis racket, bottle, wine glass, cup, fork, knife, spoon, bowl, banana, apple, sandwich, orange, broccoli, carrot, hot dog, pizza, donut, cake, potted plant, tv, laptop, mouse, remote, keyboard, cell phone, microwave, oven, toaster, book, vase, scissors, teddy bear, hair drier, toothbrush (52)</td></tr></tbody></table>

**Fig. 1: BrainHub Statistics.** We illustrate the statistics in (a) and the mapping relationships in (b) for the categories ontology used in BrainHub, w.r.t. to the original COCO classes. Please zoom in (a) for details.

# C Additional Experiments

## C.1 Brain Captioning

**More Results for S1.** In addition to the quantitative evaluation of brain captioning on S1 reported in the main paper, we report qualitative comparison in Fig. 2. As shown, SDRecon [45] often produces incoherent and irrelevant descriptions, consistent with the lowest fluency and relevance metrics in the quantitative evaluation results. OneLLM [16] provides complete and coherent responses, but typically diverges from the content in the reference visual stimuli, exhibiting the lowest CLIP similarity scores [17] (CLIP-S and RefCLIP-S). BrainCap [14] yields better results compared to the above two methods, with fluency not being an issue but encountering challenges in content similarity with the reference.

Both of our methods, UMBRAE and UMBRAE-S1, yield results with the most fluent and relevant descriptions. Here, 'UMBRAE-Sx' refers to our model trained with a single subject only and 'UMBRAE' is the unified brain decoding model with cross-subject training. When other methods fail to provide accurate
descriptions, ours can accurately describe the scenes, such as parasailing, lake, mountain in row 1; building with clock tower in row 2; bird on a tree branch in row 5, skiing in the last row. Besides, baselines tend to provide approximate descriptions in contradiction to our method, such as for giraffe in row 3, sheep in row 6 (note UMBRAE even provides accurate counts), living room in the second last. Importantly, note that unlike the brain captioning SOTAs [14, 16, 32, 45], we do *not* use any ground truth captions during the training of the brain encoder.

**Results for S2, S5, S7.** Tab. 3 shows the brain captioning results on S2, S5, S7. We compare our single-subject model UMBRAE-Sx and cross-subject model UMBRAE with two state-of-the-art methods SDRecon [45] and BrainCap [14]. Results for other subjects in OneLLM [16] and UniBrain [32] are unavailable: OneLLM [16] solely trains with S1, and UniBrain [32] has not been open-sourced, with no reported results for other subjects. The results on S2, S5, and S7 show consistent performance with those presented for S1 in the main paper. Remarkably, in all metrics and subjects, our methods perform better than the baselines [14, 45]. The models using cross-subject training (UMBRAE) also generally perform better than those trained on a single subject (UMBRAE-Sx).

**Subject Comparison.** Fig. 3 shows brain captioning results on different subjects. The first lines are ground truth captions from COCO [29] for comparison. When an image comes with multiple ground truth captions, we always select the one. The second lines display the results from Shikra [9] using images as input, which can be seen as an approximate upper bound of performance for our method. Our method shares the same adapter and the finetuned LLM with Shikra in most experiments, but uses a different encoder (brain encoder instead of image encoder) and input modality (brain response instead of image). The following lines show the results for different subjects (S1, S2, S5, and S7). Results for these subjects are all from a single UMBRAE model with subject-specific training using only the brain responses as input. Our method achieves comparable quality and relevance even when compared to ground truth, or the image captioning method using images as input.

## C.2 Brain Grounding

Results of acc@0.3 and acc@0.7 on different subject (S1, S2, S5, and S7) and different categories (A, S, SC, SO, and I) are reported in Tab. 4. Results of acc@0.5 are reported in the main paper. The IoU values remain consistent with those presented in the main paper.

Generally, the cross-subject model outperforms the single-subject models. These results are obtained using the grounding prompt `Locate <expr> in <image> and provide its coordinates, please.` where <expr> is the expression. In practice, to obtain the evaluation results shown in Tab. 4, we assume there are several 'known' concepts (<expr>) in the given image and formulate the task as detecting queried objects and returning their coordinates. This is known as the REC task in MLLMs. The example supported task prompts are shown in the second row of Tab. 1.
<figure><img src="image_7.png" /><figcaption>Fig. 2: Brain Captioning Comparison on S1. Baselines for S1 include SDRecon [45], BrainCap [14], and OneLLM [16]. ‘UMBRAE-S1’ refers to our model trained only with subject S1, while ‘UMBRAE’ denotes the model with cross-subject training.</figcaption></figure>
<table><caption>**Table 3: Brain Captioning.** 'UMBRAE-Sx' refers to our model trained with a single subject only, while UMBRAE' denotes the model with cross-subject training. The results of S1 have been presented in the main paper and are listed here for completeness. The colors represent the best, second-best, and third-best performance. Note that both our models are consistently surpassing the baselines and, furthermore, our cross-subject model (UMBRAE) is almost always better than its single-subject counterpart.</caption><thead><tr><th>Method</th><th>Eval</th><th>BLEU1</th><th>BLEU2</th><th>BLEU3</th><th>BLEU4</th><th>METEOR</th><th>ROUGE</th><th>CIDEr</th><th>SPICE</th><th>CLIP-S</th><th>RefCLIP-S</th></tr></thead><tbody><tr><td>SDRecon [44]</td><td rowspan="4">S1</td><td>36.21</td><td>17.11</td><td>7.72</td><td>3.43</td><td>10.03</td><td>25.13</td><td>13.83</td><td>5.02</td><td>61.07</td><td>66.36</td></tr><tr><td>BrainCap [14]</td><td>55.96</td><td>36.21</td><td>22.70</td><td>14.51</td><td>16.68</td><td>40.69</td><td>41.30</td><td>9.06</td><td>64.31</td><td>69.90</td></tr><tr><td>UMBRAE-S1</td><td>57.63</td><td>38.02</td><td>25.00</td><td>16.76</td><td>18.41</td><td>42.15</td><td>51.93</td><td>11.83</td><td>66.44</td><td>72.12</td></tr><tr><td>UMBRAE</td><td>59.44</td><td>40.48</td><td>27.66</td><td>19.03</td><td>19.45</td><td>43.71</td><td>61.06</td><td>12.79</td><td>67.78</td><td>73.54</td></tr><tr><td>SDRecon [45]</td><td rowspan="4">S2</td><td>34.71</td><td>15.87</td><td>6.72</td><td>3.02</td><td>9.60</td><td>24.22</td><td>13.38</td><td>4.58</td><td>59.52</td><td>65.30</td></tr><tr><td>BrainCap [14]</td><td>53.80</td><td>34.16</td><td>20.86</td><td>13.03</td><td>15.90</td><td>39.96</td><td>35.60</td><td>8.47</td><td>62.48</td><td>68.19</td></tr><tr><td>UMBRAE-S2</td><td>57.18</td><td>37.76</td><td>25.06</td><td>17.18</td><td>18.11</td><td>41.85</td><td>50.62</td><td>11.50</td><td>64.87</td><td>71.06</td></tr><tr><td>UMBRAE</td><td>59.37</td><td>40.47</td><td>27.14</td><td>18.41</td><td>19.17</td><td>43.86</td><td>55.93</td><td>12.08</td><td>66.46</td><td>72.36</td></tr><tr><td>SDRecon [45]</td><td rowspan="4">S5</td><td>34.96</td><td>16.39</td><td>7.36</td><td>3.49</td><td>9.93</td><td>24.77</td><td>13.85</td><td>5.19</td><td>60.83</td><td>66.30</td></tr><tr><td>BrainCap [14]</td><td>55.28</td><td>35.71</td><td>22.62</td><td>14.62</td><td>16.45</td><td>40.87</td><td>41.05</td><td>9.24</td><td>63.89</td><td>69.64</td></tr><tr><td>UMBRAE-S5</td><td>58.99</td><td>39.88</td><td>27.03</td><td>18.73</td><td>19.04</td><td>43.30</td><td>57.09</td><td>12.70</td><td>66.48</td><td>72.69</td></tr><tr><td>UMBRAE</td><td>60.36</td><td>41.27</td><td>27.92</td><td>19.03</td><td>20.04</td><td>44.81</td><td>61.32</td><td>13.19</td><td>68.39</td><td>74.11</td></tr><tr><td>SDRecon [45]</td><td rowspan="4">S7</td><td>34.99</td><td>16.10</td><td>7.06</td><td>3.26</td><td>9.54</td><td>24.33</td><td>13.01</td><td>4.74</td><td>58.68</td><td>64.59</td></tr><tr><td>BrainCap [14]</td><td>54.25</td><td>34.47</td><td>21.67</td><td>14.00</td><td>15.94</td><td>40.02</td><td>37.49</td><td>8.57</td><td>62.52</td><td>68.48</td></tr><tr><td>UMBRAE-S7</td><td>55.71</td><td>36.24</td><td>23.62</td><td>15.75</td><td>17.51</td><td>40.64</td><td>47.07</td><td>11.26</td><td>63.66</td><td>70.09</td></tr><tr><td>UMBRAE</td><td>57.20</td><td>38.30</td><td>25.49</td><td>17.13</td><td>18.29</td><td>42.16</td><td>52.73</td><td>11.63</td><td>65.90</td><td>71.83</td></tr></tbody></table>

**Brain Grounding without Priors.** Interestingly, our method can also provide descriptions and coordinates for brain signals without 'prior knowledge' of their contents. This task is referred to as 'Spotting Captioning' in [9], but it is a grounding-related task, as the primary goal is to describe the image (brain responses in our case) and identify the mentioned objects or regions using points or boxes. In addition to the prompts shown in Tab. 1, we can also utilize a wide range of instructions such as 'Please interpret this image and give coordinates [x1,y1,x2,y2] for each object you mention' and 'Provide a detailed description of the image using around 100-500 words, including the objects, attributes, and spatial locations depicted in the picture'. The qualitative results of brain grounding, using various task prompts and across different subjects, are provided in Fig. 4 and Fig. 5, respectively. The bounding boxes are outputted as text responses, and we visualize the outputs in the reference images. The tags `<expr>` in the REC prompt are depicted in the corresponding reference images with `color`. The concepts and coordinates in the responses of the Spotting Captioning task are depicted using the same color as the bounding box color in the visualizations.

### C.3 Brain Retrieval

Fig. 6 shows the forward and backward retrieval results using MindEye [41] and our UMBRAE. The images displayed in the top row are the reference image and the top 5 retrieval images obtained from the *forward* retrieval [28]. This process is to find the correct paired CLIP image embedding given a brain embedding. Similarly, the bottom row are the reference image and the top 5 retrieval images
<figure><img src="image_8.png" /><figcaption>**Fig. 3: Brain Captioning Results on Different Subjects.** 'COCO' is the ground truth caption in the COCO dataset [29]. We excerpt the first caption if there are multiple captions for the same image. Shikra-w/img [9] is the result using the ground truth images as input. Results for all four subjects (S1, S2, S5, and S7) are from our cross-subject UMBRAE model.</figcaption></figure>
Spotting Captioning

"Please interpret this image and give coordinates
[x₁,y₁,x₂,y₂] for each object you mention."

<figure><img src="image_9.png" /><figcaption>Fig. 4: Brain Grounding Results using Different Task Prompts. The shown results are from our UMBRAE using brain responses as input. Reference images are visual stimuli for input brain responses and are just used here for visualization. The tags `<expr>` in the REC prompt are depicted in the corresponding reference images. The bounding boxes are outputted as text responses, and we visualize the outputs in the reference images. The tags `<expr>` in the REC prompt are displayed in the reference images with `color`. The concepts and their coordinates in the Spotting Captioning responses are color-coded to match the bounding box color in the visualizations.</figcaption></figure>
<figure><img src="image_10.png" /><figcaption>**Fig. 5: Brain Grounding Result on Different Subjects.** All results are from our UMBRAE using brain responses as input. Reference images here are for visualization.</figcaption></figure>

obtained from the *backward* retrieval process, which aims to locate the correct brain embedding given an image embedding.

## C.4 Visual Decoding

Despite that our method is not specifically designed for the task of visual decoding (fMRI-to-image reconstruction), our predicted textual and visual cues can be used for the final image reconstruction by using a variety of pretrained image generation models. To be specific, we use three text-to-image models SD [40], SD-XL [37] and Kandinsky [4], a layout-to-image model GLIGEN [26], and a multiple-condition model Versatile Diffusion (VD) [55]. The quantitative evaluation results are in Tab. 5. Besides the common visual decoding metrics [34], we further evaluate the image quality using FID [18] and Clean-FID (CFID) [36], as well as the image similarity using LPIPS [57]. The qualitative comparison is illustrated in Fig. 8.

Given that our method is VD-based [55], we further analyze the effects of its inputs, as shown in the last three rows in Tab. 5. For the dual context mode (generating an image conditioned on both text and image) of VD [55], three parts can be used as inputs for the visual decoding task: the input text (t), the input image (i), and a latent code z that can either be encoded from a given image or randomly sampled. We test the upper limits of performance for different inputs. Specifically, VD-1 is using ground truth image and text, along with the latent code decoded from the ground truth image. VD-2 is similar but uses randomly sampled latent codes. VD-3 uses ground truth image and text, as well as predicted low-level images from Brain-Diffuser [34]. As shown in Tab. 5, latent codes z in the latent diffusion-based image generation models play a crucial role in the final visual decoding performance. Simply using predicted text (SD, SDXL, and Kandinsky3) or adding predicted image embedding (VD-2) is not sufficient for reliable visual decoding results. The predicted bounding boxes (GLIGEN) are
<table><caption>**Table 4: Brain Grounding Results of acc@0.3 and acc@0.7 on Different Subjects.** The accuracy with threshold *m* is abbreviated as acc@m. The IoU values remain consistent with those presented in the main paper, corresponding to each subject and category. ‘UMBRAE-Sx’ refers to our model trained with a single subject only, while ‘UMBRAE’ denotes the model with cross-subject training. Results of acc@0.5 are reported in the main paper. The best results per subject is in **color**.</caption><thead><tr><th rowspan="2">Method</th><th rowspan="2">Eval</th><th colspan="2">All (A)</th><th colspan="2">Salient (S)</th><th colspan="2">Salient Creatures (SC)</th><th colspan="2">Salient Objects (SO)</th><th colspan="2">Inconspicuous (I)</th></tr><tr><th>acc@0.3</th><th>acc@0.7</th><th>acc@0.3</th><th>acc@0.7</th><th>acc@0.3</th><th>acc@0.7</th><th>acc@0.3</th><th>acc@0.7</th><th>acc@0.3</th><th>acc@0.7</th></tr></thead><tbody><tr><td>UMBRAE-S1</td><td rowspan="2">S1</td><td>24.22</td><td>5.75</td><td>36.26</td><td>9.08</td><td>43.71</td><td>10.00</td><td>28.15</td><td>8.09</td><td>9.20</td><td>1.58</td></tr><tr><td>UMBRAE</td><td>30.47</td><td>8.47</td><td>44.45</td><td>13.55</td><td>55.86</td><td>16.14</td><td>32.04</td><td>10.73</td><td>13.01</td><td>2.14</td></tr><tr><td>UMBRAE-S2</td><td rowspan="2">S2</td><td>26.00</td><td>6.53</td><td>39.09</td><td>10.35</td><td>47.43</td><td>11.43</td><td>30.02</td><td>9.18</td><td>9.67</td><td>1.77</td></tr><tr><td>UMBRAE</td><td>29.60</td><td>7.94</td><td>42.96</td><td>12.58</td><td>55.14</td><td>16.00</td><td>29.70</td><td>8.86</td><td>12.92</td><td>2.14</td></tr><tr><td>UMBRAE-S5</td><td rowspan="2">S5</td><td>26.04</td><td>5.99</td><td>39.09</td><td>9.23</td><td>45.57</td><td>9.14</td><td>32.04</td><td>9.33</td><td>9.76</td><td>1.95</td></tr><tr><td>UMBRAE</td><td>30.05</td><td>7.28</td><td>44.75</td><td>11.47</td><td>56.14</td><td>14.29</td><td>32.35</td><td>8.40</td><td>11.71</td><td>2.04</td></tr><tr><td>UMBRAE-S7</td><td rowspan="2">S7</td><td>25.47</td><td>5.21</td><td>37.97</td><td>8.12</td><td>46.43</td><td>7.86</td><td>28.77</td><td>8.40</td><td>9.85</td><td>1.58</td></tr><tr><td>UMBRAE</td><td>28.32</td><td>7.03</td><td>42.07</td><td>10.80</td><td>53.14</td><td>12.86</td><td>30.02</td><td>8.55</td><td>11.15</td><td>2.32</td></tr></tbody></table>

<figure><img src="image_11.png" /><figcaption>**Fig. 6: Brain Retrieval Results.** We show the forward and backward retrieval results of MindEye [41] and our UMBRAE. The forward retrieval [28] (**top**) is to find the correct image embedding given a brain embedding. Conversely, the backward retrieval (**bottom**) aims to locate the correct brain embedding given an image embedding.</figcaption></figure>

intended to function as low-level constraints for the final reconstruction. However, they do not perform well on low-level metrics due to their unreliable nature.
Fig. 9 displays a qualitative comparison of individual subjects, while the quantitative evaluation of UMBRAE on all four subjects can be found in Tab. 6. Fig. 7 shows comparison on visual decoding between our method and the literature.

<figure><img src="image_12.png" /><figcaption>Fig. 7: Visual Decoding Comparison between UMBRAE and the literature on NSD. All reconstructed images are from S1.</figcaption></figure>

**Table 5: UMBRAE Visual Decoding for S1 with Various Image Generation Models.** Despite that our method is not specifically designed for fMRI-to-image reconstruction (visual decoding), the predicted textual and visual outputs can serve as cues for the final image reconstruction using a variety of pretrained image generation models. These models include text-to-image SD [40], SD-XL [37], and Kandinsky [4], a layout-to-image GLIGEN [26], and a multiple-condition Versatile Diffusion (VD) [55]. We further analyze the effects of VD's inputs in the last three rows. The colors represent the best, second-best, and third-best performance.

<table><thead><tr><th rowspan="2">Image Generation</th><th colspan="3">Low-Level</th><th colspan="3">High-Level</th><th rowspan="2">FID ↓</th><th rowspan="2">CFID ↓</th><th rowspan="2">LPIPS ↓</th></tr><tr><th>PixCorr ↑</th><th>SSIM ↑</th><th>AlexNet(2) ↑</th><th>AlexNet(5) ↑</th><th>Inception ↑</th><th>CLIP ↑</th><th>EffNet-B ↓</th><th>SwAV ↓</th></tr></thead><tbody><tr><td>SD [40]</td><td>-</td><td>.292</td><td>71.7%</td><td>83.5%</td><td>85.6%</td><td>86.1%</td><td>.786</td><td>.535</td><td>95.62</td><td>89.04</td><td>0.79</td></tr><tr><td>SDXL [37]</td><td>.070</td><td>.336</td><td>73.6%</td><td>86.9%</td><td>87.2%</td><td>86.3%</td><td>.769</td><td>.475</td><td>85.67</td><td>82.41</td><td>0.76</td></tr><tr><td>Kandinsky3 [4]</td><td>.110</td><td>.328</td><td>75.9%</td><td>85.4%</td><td>85.8%</td><td>86.4%</td><td>.789</td><td>.514</td><td>89.98</td><td>88.01</td><td>0.78</td></tr><tr><td>GLIGEN [26]</td><td>.078</td><td>.255</td><td>78.6%</td><td>90.1%</td><td>86.5%</td><td>87.5%</td><td>.766</td><td>.473</td><td>73.70</td><td>67.21</td><td>0.77</td></tr><tr><td>VD [55]</td><td>.293</td><td>.345</td><td>95.8%</td><td>97.2%</td><td>92.6%</td><td>93.9%</td><td>.690</td><td>.391</td><td>67.47</td><td>63.81</td><td>0.73</td></tr><tr><td>VD-1 (GT-ti-z)</td><td>.641</td><td>.402</td><td>99.9996%</td><td>99.9985%</td><td>99.5%</td><td>99.98%</td><td>.390</td><td>.187</td><td>54.33</td><td>44.74</td><td>0.53</td></tr><tr><td>VD-2 (GT-ti-rand-z)</td><td>.098</td><td>.244</td><td>91.0%</td><td>98.9%</td><td>98.8%</td><td>99.93%</td><td>.504</td><td>.261</td><td>59.12</td><td>53.30</td><td>0.69</td></tr><tr><td>VD-3 (GT-ti-pred-z)</td><td>.327</td><td>.352</td><td>98.975%</td><td>99.705%</td><td>98.9%</td><td>99.90%</td><td>.497</td><td>.261</td><td>62.96</td><td>52.48</td><td>0.71</td></tr></tbody></table>

**Table 6: Subject-Specific Visual Decoding Evaluation.** Quantitative evaluation of the UMBRAE reconstruction for the four subjects (S1, S2, S5, and S7) of NSD [2].

<table><thead><tr><th rowspan="2">Subject</th><th colspan="3">Low-Level</th><th colspan="3">High-Level</th><th rowspan="2">FID ↓</th><th rowspan="2">CFID ↓</th><th rowspan="2">LPIPS ↓</th></tr><tr><th>PixCorr ↑</th><th>SSIM ↑</th><th>AlexNet(2) ↑</th><th>AlexNet(5) ↑</th><th>Inception ↑</th><th>CLIP ↑</th><th>EffNet-B ↓</th><th>SwAV ↓</th></tr></thead><tbody><tr><td>S1</td><td>.293</td><td>.345</td><td>95.8%</td><td>97.2%</td><td>92.6%</td><td>93.9%</td><td>.690</td><td>.391</td><td>67.47</td><td>63.81</td><td>0.73</td></tr><tr><td>S2</td><td>.283</td><td>.353</td><td>96.2%</td><td>97.3%</td><td>90.8%</td><td>93.0%</td><td>.705</td><td>.396</td><td>70.00</td><td>66.78</td><td>0.74</td></tr><tr><td>S5</td><td>.277</td><td>.337</td><td>95.8%</td><td>97.7%</td><td>93.7%</td><td>94.8%</td><td>.689</td><td>.380</td><td>67.70</td><td>64.52</td><td>0.74</td></tr><tr><td>S7</td><td>.279</td><td>.329</td><td>95.0%</td><td>96.9%</td><td>90.5%</td><td>92.1%</td><td>.713</td><td>.405</td><td>70.07</td><td>66.50</td><td>0.75</td></tr></tbody></table>
<figure><img src="image_13.png" /><figcaption>Fig. 8: UMBRAE Visual Decoding with Various Image Generation Models.</figcaption></figure>

We use textual and visual outputs predicted by our UMBRAE model as cues for the
final image reconstruction, using a variety of pretrained image generation models. These
models include text-to-image SD [40], SD-XL [37], and Kandinsky3 [4], a layout-to-image
GIIGEN [26], and a multiple-condition VD [55]. Given that our method is based on
VD [55], we further analyze the effects of its inputs. VD-1 is using ground truth image
and text, along with the latent code decoded from the ground truth image. VD-2 is
similar but uses randomly sampled latent codes. VD-3 uses ground truth image and
text, as well as predicted low-level images from Brain-Diffuser [34].
<figure><img src="image_14.png" /><figcaption>**Fig. 9: Subject-Specific Visual Decoding Results.** Qualitative comparison of UMBRAE reconstruction for the four subjects (S1, S2, S5, and S7) of NSD [2].</figcaption></figure>

# D Additional Analyses

Here, we provide a couple of additional studies to further evaluate the per-subject performance (Appendix D.1), the weakly-supervised adaptation (Appendix D.2). We analyze different sampling strategies in cross-subject training (Appendix D.3) and present a joint grounding-decoding evaluation (Appendix D.4). We also build on MLLMs capacities to explore other tasks (Appendix D.5) and finally, demonstrate the model-agnostic characteristics of our method (Appendix D.6).

## D.1 Subject-Specific Analysis

In the main paper, we primarily present the results from two types of our models: the cross-subject training model UMBRAE and the single-subject training model UMBRAE-Sx. The former is trained using all training data from subjects {S1, S2, S5, S7}, whereas UMBRAE-Sx trains only on data from a specific subject.

Instead, here we explore training the cross-subject models with various combinations of different subjects (e.g., training on {S1, S2, S5} and testing on S1). These pretrained models are then utilized in the weakly-supervised subject adaptation experiments. The corresponding results are detailed in Tab. 7. It is interesting to note that while training with more than one subject is always improving performance, the best performance are not always achieved when using all subjects available. An example is the S7 evaluation which performs better when UMBRAE is trained only on {S1, S5, S7} (i.e., not using S2 data). We
<table><caption>**Table 7: UMBRAE Subject Analysis.** We report the per-subject evaluation of UMBRAE when varying the set of training subjects (col 'Train'). For example, 'S1' denotes our method is trained only on data from S1, i.e. this corresponds to UMBRAE-S1. Similarly, 'S1,S2,S5,S7' means training using data from all subjects, and corresponding to the model UMBRAE. 'S1,S2,S5' means the model is trained with samples from {S1, S2, S5}. 'Eval' means evaluation on a certain subject. We note that better performance are almost always achieved when training on more than one subject. The colors represent the best, second-best, and third-best performance.</caption><thead><tr><th rowspan="2">UMBRAE setting<br>Train</th><th rowspan="2">Eval</th><th colspan="7">Captioning</th><th colspan="4">Grounding</th></tr><tr><th>BLEU1</th><th>ROUGE</th><th>CIDEr</th><th>SPICE</th><th>CLIP-S</th><th>RefCLIP-S</th><th>acc@0.5 (A)</th><th>IoU (A)</th><th>acc@0.5 (S)</th><th>IoU (S)</th></tr></thead><tbody><tr><td rowspan="4">S1<br>S1,S2,S5</td><td rowspan="4">S1</td><td>57.63</td><td>42.15</td><td>51.93</td><td>11.83</td><td>66.44</td><td>72.12</td><td>13.72</td><td>17.56</td><td>21.52</td><td>25.14</td></tr><tr><td>59.75</td><td>43.53</td><td>57.36</td><td>12.79</td><td>66.39</td><td>72.63</td><td>15.63</td><td>19.30</td><td>23.60</td><td>27.15</td></tr><tr><td>58.31</td><td>42.67</td><td>55.03</td><td>12.20</td><td>65.72</td><td>71.95</td><td>14.76</td><td>18.76</td><td>22.93</td><td>26.58</td></tr><tr><td>59.23</td><td>43.70</td><td>57.00</td><td>12.69</td><td>66.25</td><td>72.42</td><td>15.58</td><td>19.34</td><td>24.05</td><td>27.58</td></tr><tr><td rowspan="4">S2<br>S1,S2,S5</td><td rowspan="4">S2</td><td>59.44</td><td>43.71</td><td>61.06</td><td>12.79</td><td>67.78</td><td>73.54</td><td>18.93</td><td>21.28</td><td>30.23</td><td>30.18</td></tr><tr><td>57.18</td><td>41.85</td><td>50.62</td><td>11.50</td><td>64.87</td><td>71.06</td><td>15.21</td><td>18.68</td><td>23.60</td><td>26.59</td></tr><tr><td>57.91</td><td>42.43</td><td>52.80</td><td>11.91</td><td>65.37</td><td>71.57</td><td>16.04</td><td>19.29</td><td>25.09</td><td>27.43</td></tr><tr><td>57.30</td><td>41.98</td><td>51.17</td><td>11.42</td><td>64.53</td><td>70.85</td><td>15.42</td><td>19.27</td><td>24.05</td><td>27.60</td></tr><tr><td rowspan="4">S2,S5,S7</td><td rowspan="4">S7</td><td>57.69</td><td>42.29</td><td>51.77</td><td>11.72</td><td>65.08</td><td>71.27</td><td>15.25</td><td>18.95</td><td>23.08</td><td>26.70</td></tr><tr><td>59.37</td><td>43.86</td><td>55.93</td><td>12.08</td><td>66.46</td><td>72.36</td><td>18.27</td><td>20.77</td><td>28.22</td><td>29.19</td></tr><tr><td>58.99</td><td>43.30</td><td>57.09</td><td>12.70</td><td>66.48</td><td>72.69</td><td>14.72</td><td>18.45</td><td>22.93</td><td>26.34</td></tr><tr><td>59.63</td><td>43.32</td><td>60.00</td><td>13.25</td><td>67.10</td><td>73.16</td><td>15.01</td><td>18.90</td><td>23.60</td><td>26.97</td></tr><tr><td rowspan="4">S5<br>S1,S2,S5</td><td rowspan="4">S5</td><td>60.02</td><td>43.50</td><td>59.67</td><td>13.31</td><td>67.24</td><td>73.39</td><td>14.84</td><td>18.68</td><td>22.86</td><td>26.34</td></tr><tr><td>59.43</td><td>43.45</td><td>59.22</td><td>12.71</td><td>67.10</td><td>73.24</td><td>15.05</td><td>18.82</td><td>23.16</td><td>26.52</td></tr><tr><td>60.36</td><td>44.81</td><td>61.32</td><td>13.19</td><td>68.39</td><td>74.11</td><td>18.19</td><td>20.85</td><td>28.74</td><td>30.02</td></tr><tr><td>55.71</td><td>40.64</td><td>47.07</td><td>11.26</td><td>63.66</td><td>70.09</td><td>13.60</td><td>17.83</td><td>21.07</td><td>25.19</td></tr><tr><td rowspan="4">S7<br>S1,S2,S7</td><td rowspan="4">S7</td><td>56.72</td><td>41.43</td><td>49.78</td><td>11.37</td><td>64.21</td><td>70.62</td><td>14.43</td><td>18.26</td><td>21.82</td><td>25.64</td></tr><tr><td>57.83</td><td>42.09</td><td>53.53</td><td>11.88</td><td>64.92</td><td>71.29</td><td>14.76</td><td>18.68</td><td>22.49</td><td>26.26</td></tr><tr><td>56.29</td><td>41.64</td><td>51.23</td><td>11.52</td><td>64.46</td><td>70.84</td><td>14.43</td><td>18.22</td><td>22.11</td><td>25.68</td></tr><tr><td>57.20</td><td>42.16</td><td>52.73</td><td>11.63</td><td>65.90</td><td>71.83</td><td>16.74</td><td>19.63</td><td>25.69</td><td>27.90</td></tr></tbody></table>

conjecture this could relate to some subjects having more similar brain activities patterns than others.

## D.2 Weakly-Supervised Adaptation

As explained in the main paper, our method enables weakly-supervised subject adaptation and can train a model for a new subject in a data-efficient manner. In other words, UMBRAE can accommodate a new subject with only a portion of the total training data. This is crucial considering the challenges in obtaining brain modality data. In the main paper, we presented the results of pretraining on {S1, S2, S5} and adapting to S7 by finetuning with varying amounts {5%, 10%, 20%, 30%, 50%, 80%, 100%} of S7 training data.

In Fig. 10, we present the adaptation to other subjects, being: **Fig. 10a** training on {S2, S5, S7}, adaptation to S1; **Fig. 10b** training on {S1, S5, S7}, adaptation to S2; **Fig. 10c** training on {S1, S2, S7}, adaptation to S5. In the above mentioned figures, we report performance when finetuning both the tokenizer and the encoder, which was proven to provide the best performance in the main paper. As for adaptation to S7 (cf. main paper), compared to the single-subject model 'UMBRAE-Sx', our 'Finetuned' adaptation on S1, S2, and S5 achieves comparable performance using only 30% of the data.
Captioning

<figure><img src="image_15.png" /><figcaption>(a) Adaptation to S1 (trained on S2, S5, and S7)</figcaption></figure>

<figure><img src="image_16.png" /><figcaption>(b) Adaptation to S2 (trained on S1, S5, and S7)</figcaption></figure>

<figure><img src="image_17.png" /><figcaption>(c) Adaptation to S5 (trained on S1, S2, and S7)</figcaption></figure>

Grounding

<figure><img src="image_18.png" /></figure>

<figure><img src="image_19.png" /></figure>

<figure><img src="image_20.png" /></figure>

**Fig. 10: Weakly-Supervised Subject Adaptation.** We plot our performance for adaptation to a given subject (S1 in **a**, S2 in **b**, S5 in **c**). For adapting to a given subject Sx, we first train UMBRAE on all other subjects and then finetune the pretrained model with varying ratios of training data from the target subject.
<figure><img src="image_21.png" /></figure>

<table><caption>**Table 8: Comparison of Sampling Strategies.** 'Random' means all subjects are randomly sampled, while 'Stratified' ensures that data from the four subjects are equal in number within a batch. 'Ours-R' and 'Ours' are the same when sampling from the selected dominant subject but differ for the remaining three subjects.</caption><thead><tr><th rowspan="2">Sampling</th><th colspan="4">Captioning</th><th colspan="4">Grounding</th></tr><tr><th>BLEU1</th><th>CIDEr</th><th>METER</th><th>RefCLIPS</th><th>acc@0.5 (A)</th><th>IoU (A)</th><th>acc@0.5 (S)</th><th>IoU (S)</th></tr></thead><tbody><tr><td>Random</td><td>51.84</td><td>35.37</td><td>15.52</td><td>67.26</td><td>13.49</td><td>17.39</td><td>21.56</td><td>25.39</td></tr><tr><td>Stratified</td><td>58.91</td><td>55.83</td><td>18.94</td><td>72.69</td><td>17.31</td><td>20.34</td><td>27.08</td><td>29.03</td></tr><tr><td>Ours-R</td><td>58.02</td><td>55.02</td><td>18.68</td><td>72.09</td><td>15.22</td><td>18.84</td><td>23.58</td><td>26.74</td></tr><tr><td>Ours</td><td><strong>59.09</strong></td><td><strong>57.76</strong></td><td><strong>19.24</strong></td><td><strong>72.96</strong></td><td><strong>18.03</strong></td><td><strong>20.63</strong></td><td><strong>28.22</strong></td><td><strong>29.32</strong></td></tr></tbody></table>

## D.3 Sampling Strategies in Cross-Subject Training

There are different sampling strategies of subjects and data samples in the cross-subject training. Our batch of *B* samples is made of $\theta \times B$ samples from the same user chosen according to users frequencies, while the remaining samples are then sampled from other users. This is illustrated on the left of Tab. 8 using, for simplicity, four users, $\theta=0.44$ and $B=16$. 'Random' means all subjects are randomly sampled, while 'Stratified' ensures that data samples from the four subjects are equal in number within a batch. 'Ours-R' and 'Ours' are the same for the dominant subject but differ in the sampling strategies for the remaining three subjects. Using a dominant subject per batch helps the model to learn intra-subject variations while being exposed to other subjects patterns to enhance inter-subject discrimination and alleviate catastrophic forgetting. Tab. 8 reports average metrics across users for 'Random', 'Stratified', 'Ours-R', and 'Ours' (using $\theta=0.50$, $B=256$). Ours outperforms all other sampling strategies.

## D.4 Joint Grounding-Decoding Evaluation

We visualize grounding results on the reference images (ground truth) to better assess their performance. Fig. 11 further shows grounding and reconstruction simultaneously to highlight their synergy. Results demonstrate that the two tasks are correlated. In some cases, although grounding is correct, reconstruction is inaccurate (e.g. surfer, giraffe, and skier are well located but misoriented). The first row presents reference images. The second row displays reconstructed images, which are generated using the decoded texts and groundings from the third row as inputs. The third row illustrates the spotting captioning results, where the coordinates for each mentioned object are omitted and instead visualized in color within the corresponding generated images shown in the second row.

## D.5 Other Supported Tasks

As we build on MLLM, we can explore a large variety of tasks. Tab. 1 lists the supported tasks, which can be categorized into three groups: captioning, grounding, and QA. We have presented the brain captioning results in Appendix C.1 and the brain grounding (both REC and Spotting Captioning) results in Appendix C.2.
<figure><img src="image_22.png" /><figcaption>**Fig. 11: Joint Visualization of Grounding and Reconstruction.** The first row presents reference images. The second row displays reconstructed images generated using decoded texts and groundings from the third row. The coordinates in the third row for each mentioned object are omitted and visualized in color in the second row. Reconstruction may be inaccurate even with correct bounding boxes.</figcaption></figure>

This section presents the additional QA tasks, including $Q \to A$, $Q \to CA$, and $Q \to C^{\text{Box}}A$. Example results are shown in Fig. 12.

## D.6 Model-Agnostic Performance

We employ the pretrained adapter and LLM from Shikra [9] due to its simultaneous support for both captioning and grounding. However, UMBRAE being model-agnostic, it can accommodate any image encoders (e.g., CLIP [39] and DINO [7]), LLMs (e.g., LLaMA [48] and Vicuna [11]), and MLLMs (e.g., Shikra [9] and LLaVA [30]) according to specific needs.

To demonstrate this capacity, in Fig. 13, we provide qualitative results of ‘UMBRAE with LLaVA’ which replaces Shikra with the nonetheless popular LLaVA [30] as the MLLM. These example responses show that LLaVA [30] exhibits superior instruction-following capabilities in conversation, detailed description, and complex reasoning compared to Shikra [9]. However, LLaVA [30] lacks the capability of spatial localization.

# E Discussion

## E.1 Method Limitations

Our goal is to decode brain signals into multimodal explanations, with a primary emphasis on semantics and positions, achieving both concept recognition and spatial localization. While results demonstrate decent performance, there are instances where it falls short. Apart from previously mentioned challenges with inconspicuous objects, other failure cases can be categorized into three types: (a) correct semantic recognition but inaccurate spatial localization, (b) accurate spatial localization but semantic errors, and (c) errors in both semantic recognition
Describe this image as simply as possible.

A man riding a wave on top of a surfboard.

What is the weather like? How should I dress accordingly for the weather?

The answer is sunny.

What are the things I should be cautious about?

The image shows a man wearing a wetsuit, standing on a surfboard, and riding a wave in the ocean. As a surfer, there are several factors to be cautious about while engaging in this activity. These include:

<figure><img src="image_23.png" /></figure>

1. **Wave conditions:** Always pay attention to the size, power, and frequency of the waves. Understanding and predicting wave behavior is crucial for a successful surfing experience.

2. **Weather conditions:** Be aware of the weather, including wind speed and direction, which can influence wave formation and quality.

3. **Ocean currents and tides:** Be mindful of strong currents and tides, which can quickly pull a surfer away from the shore or make it difficult to return to the shore.

4. **Other surfers and water users:** Always maintain a safe distance from other surfers, swimmers, or watercraft to avoid collisions and accidents.

5. **Equipment:** Ensure that the surfboard is suitable for the individual and the current conditions, and that it is in good condition, free from any damage that may affect its performance.

6. **Safety gear:** Wearing a wetsuit, as seen in the image, can protect against cold water temperatures, potential hazards like rocks or coral reefs, and potential injuries from the surfboard or sharp objects in the water.

By being aware of these factors and taking necessary precautions, surfers can have a safer and more enjoyable experience in the ocean.

Having a look at image , can you tell me the answer to my question “*is the man in danger*” and the logic leading to it?

A man surfing in the ocean on a large wave. So the answer is no.

Analyze the image and answer "what is the man doing". Include your reasoning process and provide bounding boxes of related objects g [x0,y0,x1,y1].

The man [0.426,0.280,0.560,0.628] is wearing a wetsuit [0.418,0.316,0.566,0.570] and is standing on a surfboard [0.320,0.616,0.630,0.654] while riding a wave [0.002,0.438,0.998,0.996] in the ocean [0.002,0.008,0.998,0.998]. This indicates that he is surfing.

<figure><img src="image_24.png" /></figure>

**Fig. 12: Example Results of Other Tasks.** The additional supported tasks listed in Tab. 1 include $Q \to A$, $Q \to CA$, and $Q \to C^{\text{Box}}A$.

and spatial localization. We use spotting captioning to illustrate failure cases in Fig. 14, as this task outputs both concepts and locations.

Moreover, our method relies on pretrained MLLMs, inheriting their advantages while also facing common shortcomings associated with large models, including biases, hallucinations, generation of inappropriate content, and potential ethical concerns. Our method is also constrained by the quality of the captured brain responses in NSD [2] in two ways. Firstly, there are inherent inaccuracies introduced during data collection. NSD is captured using non-invasive neuroimaging techniques, where participants' compliance is necessary to avoid disruption in decoding caused by head movement or distraction. Secondly, the experimental images are sourced from COCO [29], which limits our method to natural scenes similar to those found in the COCO dataset [29].

## E.2 Potential Negative Impact

Our method relies on pretrained models as its foundation. While benefiting from the remarkable capabilities provided by LLMs [9, 16, 48], they also pose challenges
<figure><img src="image_25.png" /></figure>

The weather in the image appears to be sunny and clear, with no visible clouds or signs of rain. The water looks clear, suggesting a pleasant day for surfing.

For such weather, you should dress appropriately for water sports. This would typically include:

1. **Wetsuit:** If the water temperature is cool, you would need a wetsuit to keep warm. The thickness of the wetsuit depends on the water temperature.

2. **Swimwear:** If the water is warm enough, you can wear swimwear, such as a swimsuit or swim shorts.

3. **Sun Protection:** Even on a sunny day, the sun can be intense, so it's important to protect your skin from sunburn. Wear sunscreen with a high SPF, a hat, and sunglasses.

4. **Surf Accessories:** Depending on your surfing level, you might need additional accessories like a surf leash, surf wax, or a rash guard to protect your skin from the wax and friction of the surfboard.

5. **Shoes:** Some surfers prefer to wear water shoes to protect their feet from sharp rocks or reef cuts.

Always check the local weather forecast and water temperature before heading out to surf to ensure you are dressed appropriately for the conditions.

### What are the things I should be cautious about?

In the image, you see a person surfing on a wave. When surfing, there are several things to be cautious about:

1. **Water Conditions:** Be aware of the water conditions, including the size and frequency of the waves, the presence of rip currents, and the water temperature.

2. **Surboard:** Ensure that your surfboard is in good condition and that you are using appropriate safety gear, such as a leash to keep the board attached to your ankle.

3. **Surfing Skills:** Know your limits and don't attempt to surf waves that are beyond your skill level.

4. **Environmental Hazards:** Be mindful of any environmental hazards in the area, such as rocks, reefs, or marine life.

5. **Other Surfers:** Be respectful of other surfers in the water, maintaining a safe distance and taking turns catching waves.

6. **Weather Conditions:** Check the weather forecast before surfing, as sudden changes in weather can make the conditions dangerous.

7. **Safety Measures:** Learn basic first aid and know how to handle emergencies, such as injuries or getting caught in a rip current.

8. **Local Regulations:** Familiarize yourself with local surfing regulations, which may include restrictions on surfing in certain areas or during specific times.

9. **Environmental Impact:** Be conscious of your impact on the environment, such as not littering and respecting the local ecosystem.

10. **Physical Fitness:** Surfing requires a certain level of physical fitness, so make sure you are in good shape before attempting to surf.

Always prioritize safety and respect for the environment when engaging in water sports like surfing.

<figure><img src="image_26.png" /><figcaption>**Fig. 14:** Method Limitation. The failure cases can be categorized into: (a) correct semantic recognition but inaccurate spatial localization; (b) accurate spatial localization but semantic errors; (c) errors in both semantic recognition and spatial localization.</figcaption></figure>

and concerns that prompt broader societal impacts. These include potential biases in the training data, the generation of inaccurate or inappropriate content, and ethical considerations associated with their utilization. The inaccurate interpretation from our method may also lead to misunderstandings about the information contained within brain signals.
References

1. Admoni, H., Scassellati, B.: Social eye gaze in human-robot interaction: a review. Journal of Human-Robot Interaction **6**(1), 25–63 (2017)

2. Allen, E.J., St-Yves, G., Wu, Y., Breedlove, J.L., Prince, J.S., Dowdle, L.T., Nau, M., Caron, B., Pestilli, F., Charest, I., et al.: A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence. Nature neuroscience **25**(1), 116–126 (2022)

3. Anderson, P., Fernando, B., Johnson, M., Gould, S.: Spice: Semantic propositional image caption evaluation. In: ECCV. pp. 382–398. Springer (2016)

4. Arkhipkin, V., Filatov, A., Vasilev, V., Maltseva, A., Azizov, S., Pavlov, I., Aga-fonova, J., Kuznetsov, A., Dimitrov, D.: Kandinsky 3.0 technical report. arXiv preprint arXiv:2312.03511 (2023)

5. Banerjee, S., Lavie, A.: Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In: ACL Workshop. pp. 65–72 (2005)

6. Caron, M., Misra, I., Mairal, J., Goyal, P., Bojanowski, P., Joulin, A.: Unsupervised learning of visual features by contrasting cluster assignments. NeurIPS **33**, 9912–9924 (2020)

7. Caron, M., Touvron, H., Misra, I., Jégou, H., Mairal, J., Bojanowski, P., Joulin, A.: Emerging properties in self-supervised vision transformers. In: ICCV. pp. 9650–9660 (2021)

8. Chaudhary, U., Vlachos, I., Zimmermann, J.B., Espinosa, A., Tonin, A., Jaramillo-Gonzalez, A., Khalili-Ardali, M., Topka, H., Lehmberg, J., Friehs, G.M., et al.: Spelling interface using intracortical signals in a completely locked-in patient enabled via auditory neurofeedback training. Nature communications **13**(1), 1236 (2022)

9. Chen, K., Zhang, Z., Zeng, W., Zhang, R., Zhu, F., Zhao, R.: Shikra: Unleashing multimodal llm's referential dialogue magic. arXiv preprint arXiv:2306.15195 (2023)

10. Chen, Z., Qing, J., Zhou, J.H.: Cinematic mindscapes: High-quality video reconstruction from brain activity. Advances in Neural Information Processing Systems **36** (2024)

11. Chiang, W.L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J.E., et al.: Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. 2023. URL https://lmsys.org/blog/2023-03-30-vicuna/1(2), 3 (2023)

12. Cui, Y., Zhang, Q., Knox, B., Allievi, A., Stone, P., Niekum, S.: The empathic framework for task learning from implicit human feedback. In: CoRL. pp. 604–626. PMLR (2021)

13. Desimone, R., Albright, T.D., Gross, C.G., Bruce, C.: Stimulus-selective properties of inferior temporal neurons in the macaque. Journal of Neuroscience **4**(8), 2051–2062 (1984)

14. Ferrante, M., Ozcelik, F., Boccato, T., VanRullen, R., Toschi, N.: Brain captioning: Decoding human brain activity into images and text. arXiv preprint arXiv:2305.11560 (2023)

15. Girdhar, R., El-Nouby, A., Liu, Z., Singh, M., Alwala, K.V., Joulin, A., Misra, I.: Imagebind: One embedding space to bind them all. In: CVPR (2023)

16. Han, J., Gong, K., Zhang, Y., Wang, J., Zhang, K., Lin, D., Qiao, Y., Gao, P., Yue, X.: Onellm: One framework to align all modalities with language. In: CVPR (2024)

17. Hessel, J., Holtzman, A., Forbes, M., Bras, R.L., Choi, Y.: Clipscore: A reference-free evaluation metric for image captioning. In: EMNLP (2021)
18. Heusel, M., Ramsauer, H., Unterthiner, T., Nessler, B., Hochreiter, S.: Gans trained by a two time-scale update rule converge to a local nash equilibrium. NeurIPS 30 (2017)

19. Hussein, A., Gaber, M.M., Elyan, E., Jayne, C.: Imitation learning: A survey of learning methods. CSUR 50(2), 1–35 (2017)

20. Jaegle, A., Gimeno, F., Brock, A., Vinyals, O., Zisserman, A., Carreira, J.: Perceiver: General perception with iterative attention. In: ICML. pp. 4651–4664. PMLR (2021)

21. Kanwisher, N., McDermott, J., Chun, M.M.: The fusiform face area: a module in human extrastriate cortex specialized for face perception. Journal of neuroscience 17(11), 4302–4311 (1997)

22. Karras, T., Laine, S., Aittala, M., Hellsten, J., Lehtinen, J., Aila, T.: Analyzing and improving the image quality of StyleGAN. In: CVPR. pp. 8107–8116 (2020)

23. Krizhevsky, A., Sutskever, I., Hinton, G.E.: Imagenet classification with deep convolutional neural networks. Communications of the ACM 60(6), 84–90 (2017)

24. Laureys, S., Pellas, F., Van Eeckhout, P., Ghorbel, S., Schnakers, C., Perrin, F., Berre, J., Faymonville, M.E., Pantke, K.H., Damas, F., et al.: The locked-in syndrome: what is it like to be conscious but paralyzed and voiceless? Progress in brain research 150, 495–611 (2005)

25. Lee, S., Zhang, R., Hwang, M., Hiranaka, A., Wang, C., Ai, W., Tan, J.J.R., Gupta, S., Hao, Y., Levine, G., et al.: Noir: Neural signal operated intelligent robots for everyday activities. In: CoRL (2023)

26. Li, Y., Liu, H., Wu, Q., Mu, F., Yang, J., Gao, J., Li, C., Lee, Y.J.: Gligen: Open-set grounded text-to-image generation. In: CVPR (2023)

27. Lin, C.Y.: Rouge: A package for automatic evaluation of summaries. In: Text summarization branches out. pp. 74–81 (2004)

28. Lin, S., Sprague, T., Singh, A.K.: Mind Reader: Reconstructing complex images from brain activities. NeurIPS 35, 29624–29636 (2022)

29. Lin, T.Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ramanan, D., Dollár, P., Zitnick, C.L.: Microsoft COCO: Common objects in context. In: ECCV. pp. 740–755 (2014)

30. Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning. NeurIPS 36 (2023)

31. Loshchilov, I., Hutter, F.: Decoupled weight decay regularization. In: ICLR (2019)

32. Mai, W., Zhang, Z.: Unibrain: Unify image reconstruction and captioning all in one diffusion model from human brain activity. arXiv preprint arXiv:2308.07428 (2023)

33. Oord, A.v.d., Li, Y., Vinyals, O.: Representation learning with contrastive predictive coding. arXiv preprint arXiv:1807.03748 (2018)

34. Ozcelik, F., VanRullen, R.: Brain-Diffuser: Natural scene reconstruction from fMRI signals using generative latent diffusion. Scientific Reports 13(1), 15666 (2023)

35. Papineni, K., Roukos, S., Ward, T., Zhu, W.J.: Bleu: a method for automatic evaluation of machine translation. In: ACL. pp. 311–318 (2002)

36. Parmar, G., Zhang, R., Zhu, J.Y.: On aliased resizing and surprising subtleties in gan evaluation. In: CVPR. pp. 11410–11420 (2022)

37. Podell, D., English, Z., Lacey, K., Blattmann, A., Dockhorn, T., Müller, J., Penna, J., Rombach, R.: Sdxl: Improving latent diffusion models for high-resolution image synthesis. In: ICLR (2024)

38. Puce, A., Allison, T., Asgari, M., Gore, J.C., McCarthy, G.: Differential sensitivity of human visual cortex to faces, letterstrings, and textures: a functional magnetic resonance imaging study. Journal of neuroscience 16(16), 5205–5215 (1996)

39. Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: ICML. pp. 8748–8763. PMLR (2021)
40. Rombach, R., Blattmann, A., Lorenz, D., Esser, P., Ommer, B.: High-resolution image synthesis with latent diffusion models. In: CVPR. pp. 10684–10695 (2022)

41. Scotti, P.S., Banerjee, A., Goode, J., Shabalin, S., Nguyen, A., Cohen, E., Dempster, A.J., Verlinde, N., Yundler, E., Weisberg, D., et al.: Reconstructing the mind's eye: fmri-to-image with contrastive learning and diffusion priors. In: NeurIPS (2023)

42. Smith, L.N., Topin, N.: Super-convergence: Very fast training of neural networks using large learning rates. In: Artificial intelligence and machine learning for multi-domain operations applications. vol. 11006, pp. 369–386. SPIE (2019)

43. Szegedy, C., Vanhoucke, V., Ioffe, S., Shlens, J., Wojna, Z.: Rethinking the inception architecture for computer vision. In: CVPR. pp. 2818–2826 (2016)

44. Takagi, Y., Nishimoto, S.: High-resolution image reconstruction with latent diffusion models from human brain activity. In: CVPR. pp. 14453–14463 (2023)

45. Takagi, Y., Nishimoto, S.: Improving visual image reconstruction from human brain activity using latent diffusion models via multiple decoded inputs. arXiv preprint arXiv:2306.11536 (2023)

46. Tan, M., Le, Q.: Efficientnet: Rethinking model scaling for convolutional neural networks. In: ICML. pp. 6105–6114. PMLR (2019)

47. Tang, J., LeBel, A., Jain, S., Huth, A.G.: Semantic reconstruction of continuous language from non-invasive brain recordings. Nature Neuroscience pp. 1–9 (2023)

48. Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

49. Uddin, L.Q.: Salience processing and insular cortical function and dysfunction. Nature reviews neuroscience 16(1), 55–61 (2015)

50. Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A.N., Kaiser, Ł., Polosukhin, I.: Attention is all you need. NeurIPS 30 (2017)

51. Vedantam, R., Lawrence Zitnick, C., Parikh, D.: Cider: Consensus-based image description evaluation. In: CVPR. pp. 4566–4575 (2015)

52. Wang, Z., Bovik, A.C., Sheikh, H.R., Simoncelli, E.P.: Image quality assessment: from error visibility to structural similarity. TIP 13(4), 600–612 (2004)

53. Xia, W., de Charette, R., Öztireli, C., Xue, J.H.: Dream: Visual decoding from reversing human visual system. In: WACV. pp. 8226–8235 (2024)

54. Xia, W., Yang, Y., Xue, J.H., Feng, W.: Controllable continuous gaze redirection. In: ACM MM. pp. 1782–1790 (2020)

55. Xu, X., Wang, Z., Zhang, E., Wang, K., Shi, H.: Versatile diffusion: Text, images and variations all in one diffusion model. In: ECCV. pp. 7754–7765 (2023)

56. Yu, L., Poirson, P., Yang, S., Berg, A.C., Berg, T.L.: Modeling context in referring expressions. In: ECCV. pp. 69–85. Springer (2016)

57. Zhang, R., Isola, P., Efros, A.A., Shechtman, E., Wang, O.: The unreasonable effectiveness of deep features as a perceptual metric. In: CVPR. pp. 586–595 (2018)