MindLLM: A Subject-Agnostic and Versatile Model for fMRI-to-text Decoding
Authors: Weikang Qiu
Zheng Huang
Haoyu Hu
Aosong Feng
Yujun Yan
Rex Ying
Venue: ICML 2025 poster (2025)
Keywords: Neuroscience
fMRI
Primary area: applications->neuroscience_cognitive_science
TL;DR: we propose a Subject-Agnostic and Versatile Model for fMRI-to-text Decoding

Abstract: Decoding functional magnetic resonance imaging (fMRI) signals into text has been a key challenge in the neuroscience community, with the potential to advance brain-computer interfaces and uncover deeper insights into brain mechanisms. However, existing approaches often struggle with suboptimal predictive performance, limited task variety, and poor generalization across subjects. In response to this, we propose MindLLM, a model designed for subject-agnostic and versatile fMRI-to-text decoding. MindLLM consists of an fMRI encoder and an off-the-shelf LLM. The fMRI encoder employs a neuroscience-informed attention mechanism, which is capable of accommodating subjects with varying input shapes and thus achieves high-performance subject-agnostic decoding. Moreover, we introduce Brain Instruction Tuning (BIT), a novel approach that enhances the model's ability to capture diverse semantic representations from fMRI signals, facilitating more versatile decoding. We evaluate MindLLM on comprehensive fMRI-to-text benchmarks. Results demonstrate that our model outperforms the baselines, improving downstream tasks by $12.0\%$, unseen subject generalization by $24.5\%$, and novel task adaptation by $25.0\%$. Furthermore, the attention patterns in MindLLM provide interpretable insights into its decision-making process.

MindLLM: A Subject-Agnostic and Versatile Model for fMRI-to-TTxx Decoding
Weikang Qiu Zheng Huang Haoyu Hu"
Aosong Feng'
Yujun Yan 2 Rex Ying'
Abstract
Sunjezte
Perseption
Decoding functional magnetic resonance imaging (fMRI) signals into text has been key challenge in the neuroscience community, with the potential to advance brain-computer interfaces and uncover deeper insights into brain mechanisms. However, existing approaches often struggle with suboptimal predictive performance, limited task variety, and poor generalization across subjects. In response to this, we propose MindLLM, model designed for subject-agnostic and versatile fMRI-to-text decoding. MindLLM consists of an fMRI encoder and an off-the-shelf LLM. The
Novery & Krowletge Raytovel
Symbblii Proseesing
Training
Usage
Mante to turn off the Light.
Downatrears Apprication
Erenader
Emboaded
Moxi..M
do right nourt
Oyen
Quaction
IMRI encoder employs a neuroscience-infomee attention mechanism, which is capable of accom- modating subjects with varying input shapes and thus achieves high-performance subject-agnostic decoding. Moreover, we introduce Brain Instruc- tion Tuning (BIT), novel approach that enhances the model's ability to capture diverse semantic representations from IMRI signals, facilitating more versatile decoding. We evaluate MindLLM comprehensive fMRI-to-text benchmarks. Results demonstrate that our model outperforms the baselines, improving downstream tasks by 12.0%, unseen subject generalization by 24.5%. and novel task adaptation by 25.0%. Furthermore, the attention patterns in MindLLM provide inter- pretable insights into its decision-making process. Code is available https:: //ttt::://ttthbb. com/ Craph-and- wewndd--eomerrc--eeeaeteeee MindLIM.
Figure I. The overview of method. MindLLM is equipped with subject-agnostic 1MRI encoder and an off the-shelf LLM. MindLLM trained on multiple subjects with varying input shapes and an instruction-tuning dataset, aiming to encode different facets of semantic information in IMRI. After training, MindLLM capable of various text decoding tasks. One application is that the decoded contents can be used achieve neural control of existing systems that are not designed for it.
Hmamouche et al., 2024). The ability to translate brain activity patterns into natural language carries both academic and societal importance. For neuroscientists, it provides deeper and novel insights into cognition, behavior, and con- sciousness (Qiu et al., 2023; Luo et al., 2023), On societal level, it presents opportunities for medical applications and improves human-computer interaction (HCI) (Bernal et al., 2022; Du et al., 2022). For example, for individuals with speech impairments, this technology could restore commu- nication capabilities, enabling them to express their thoughts effortlessly (Card al., 2024). Moreover, as shown in Fig- ure 1, it benefits healthy individuals by allowing neural con- trol of digital devices, such as embodied AIs or prosthetic limbs, allowing for more intuitive and precise movements.
1. Introduction
Decoding human brain activity (fMRI) to text has sparked significant interest within the neuroscience community (Xia et al., 2024; Chen et al., 2023a: Luo et al., 2023;
Despite potential, decoding brain activity to language still faces significant challenges. One major obstacle the need for versatile decoding tailored to specific applications. For example, decoding may aim to translate a subject's movement intention to control a prosthesis, or to interpret abstract thoughts or memories. Traditional models fail to accommodate such diverse requirements. To address this, UMBRAE (Xia et al., 2024) integrates a Visual Language Model (VLM) (Chen et al., 2023b) and learns to map from
'Equal contribution Yale University DDatmoutt College *University Cambridge. Correspondence to: Weikang Qiu <weikang qiuu@yaleeddu..
Proceedings of the 4dnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).

---

MindLLM: A Subject-Agnostic and Versatile Model for IMRI-to-Texi Decoding
fMRI data corresponding stimulus images. While this approach achieves versatility to some extent, it remains constrained to tasks directly tied to the current stimulus image and cannot address broader tasks, such as retrieving memories of past visual experiences.
%
Tunen
Sample
Neurozoienaa Informed Albention
To Chw..ndaptt..o Eos
Another critical challenge lies in designing a unified and subject-agnostic architecture. Current methods of brain multimodal decoding mostly rely on a preprocessing step: selecting responsive voxels by comparing the baske fMRI to the resting-state fMRI. The selection typically re- sults in higher performance compared to using whole-brain data. However, the varying number and irregular spatial dis- tribution of selected voxels across subjects pose significant challenges for developing a unified architecture. To this end, recent studies (Wang et al., 2024a;b) have proposed pooling or sampling voxels to standardize input dimensions. However, as illustrated in Figure 3, these methods still suffer from the loss of spatial information and uneven represen- tations of certain areas, ultimately compromising perfor- mance.
(a) MindBeidge
(b) UniBazin
[c] MindLLM (our)
Figure 2. Comparison between our model and previous unified models. MindBridge (Wang et al., 2024a) flattens the voxels and adaptively pools them to a fixed dimension, which overlooks the rich information in positions. UniBrain (Wang et al., 2024b) uni- formly samples subset of voxels and aggregates their neighbors. Different from these methods, we propose sesuroscienceinfornee attention, where each query token attends to all voxels, which minimizes potential information loss in pooling or sampling.
2. Related Works
Drain-Conditioned Text Generation
This line of research mostly focuses decoding perceived visual stimuli into natural language from fMRI signals. MindGPT (Chen et al., 2023a), UniBrain(Mai & Zhang, 2023) and BrainCap (Ferrante el al., 2023) employ an IMRI encoder guided by CLIP (Radford et al., 2021) and use a language model (Radford et al., 2019; Wang et al., 2022) to decode natural language from the encoded representations. BrainChat (Huang, 2024) utilizes multiple pretraining strate- gies (Devlin, 2018; He et al., 2022; Yu et al., 2022) to align fMRI with image and text embeddings. These methods fall short in performance and versatility. UMBRAE (Xia et al., 2024) proposes to learn a mapping from fMRI to stimulus images, which later serves as a proxy input for an off-the- shelf visual language model (VLM). Although they achieve performance improvements, the strategy prevents the model from performing tasks that are not directly related to the stimulus images (e.g., answering memory related questions). In contrast, our end-to-end Brain Instruction Tuning (BIT) ensures seamless and versatile fMRI-to-text decoding, of- fering the potential to tackle tasks beyond vision-related ones.
Present Work Here we present MindLLM, a subject- agnostic and versatile model for IMRI-to-text decoding. Our approach consists of subject-agnostic fMRI encoder and an off-the-shelf LLM. The subject-agnostic fMRI en- coder incorporates a cnuroociennc-informen attention layer with learnable queries, enabling dynamic feature extraction by leveraging both spatial information and ncuroscientific priors of voxels, thereby significantly enhancing predic- tion accuracy. The design of values and keys separates the is largely consistent
voxel's functional information across individuals-from its fMRI value, allowing the model to benefit from shared priors across subjects and enhancing generalization to novel subjects. Moreover, to address the challenge of versatile decoding, we propose Brain Instruc- tion Tuning (BIT). BIT trains the model using a diverse dataset that employs images as intermediaries, encompass- ing tasks designed to capture diverse aspects of semantic information encoded in fMRI data, including perception & scene understanding, memory & knowledge retrieval, language & symbolic processing, and complex reasoning. Figure illustrates the corresponding components.
Cross-subjects Decoding In voxel-level machine learning for brain decoding, the number of voxels varies between subjects (Allen et al., 2022). Most prior works (Scotti et al., 2024a;b) use an MLP for each subject individually. How- ever, due to the fixed input size required by MLP architec- tures, these models cannot handle varying input shapes. As illustrated in Figure 2, MindBridge (Wang et al., 2024a) proposed to use an adaptive max pooling layer to standard- ize the input shapes. However, unlike images, which are considered translation invariance, positions in fMRI carry specific bio-informative significance that pooling operations may overlook. UniBrain (Wang et al., 2024b) proposed to sample groups of voxels. Such a sampling strategy, on the one hand, may lead to information loss if some voxels are
We evaluate our model on comprehensive benchmarks. Re- sults reveal it outperforms baselines with 12.0% average improvement in various downstream tasks and 24.5% im- provement in generalization on unseen subjects. Addition- ally, we show that our model adapts effectively to novel tasks, demonstrating high customizability and flexibility in real-world applications. Furthermore, our analysis of at- tention weights offers valuable insights into the working mechanism of our fMRI encoder.
2

---

MindLLM: A Subject-Agnostic and Versatile Model for IMRI-to-Texl Decoding
not included in any group. On the other hand, the irregular spatial distribution of 31D voxels with varying density and curvature may result in underrepresentation overrepresen- tation of certain areas. Different from these methods, our model employs a neuroscience-informees attention mech- anism that accounts for every single voxel while preserv- ing their bio-informative positional information, ensuring a more comprehensive and precise representation.
Vicuna-7b (Zheng et al., 2023) as our LLM maintain consistency with our baseline (Xia al., 2024). For each sample, let v = [Ui, "2,, ,NN] E IR- be the fMRI signals of input voxels, where N is the number of voxels. Note that N varies between different subjects, ranging from 12, 682 to 17, 907 in the dataset we use (Allen et al., 2022).
The fMRI encoder fe. featuring a neuroscience-infommee attention layer. encodes v to fMRI tokens X 1, Fu,2, ... -Lr R where L is the number of tokens and d is the dimension of token embeddings. We then prepend these learned fMRI tokens to the language tokens in the BIT dataset we propose.
Multi-Modal Large Language Model Aiming to aug- ment the perceptual capacities of Large Language Models (LLMs), there has been a growing interest in extending them to handle multiple modalities within a unified model. Nu- merous studies have attempted to incorporate modalities such as images (Alayrac et al., 2022; Zhang et al., 2023b; Wang et al.,, 2023b), videos (Cheng et al., 2024; Kondratyuk et al., 2023; Zhang et al., 2023a), and point clouds (Xu et al., 2023; Qi et al., 2025). OneLLLM (Han et al., 2024a) stands out by aligning eight different modalities, including fMRI, with language. However, their approach employs an individ- ual convolution network for each subject instead of a uni- fied architecture for fMRI encoding across subjects, which restricts its applicability to new subjects in real-world sce- narios. Furthermore, the approach solely relies on captions as textual annotations, which limits the model's capability for versatile fMRI decoding.
3.2. IMRI Encoder
As mentioned before, currently most models for fMRI de- coding can not handle varying input shapes and are not subject-agnostic, with only a few exceptions (Wang et al., 2024b). However, these exceptions still suffer from informa- tion loss and uneven representations of certain brain areas. To this end, we propose novel ceenssiieneeinffomme at- tention mechanism to accommodate varying voxel numbers across subjects, enabling subject-agnostic encoding strat- egy. Below we talk about the design of queries {q.), keys {k:} and values {v:} in the attention layer. For values, we directly use the fMRT signal of each voxel, which means Vs- = E R. Making each voxel value token maximally prevents information loss compared to pooling- (Wang et al., based (Mai & Zhang, 2023) methods. The queries are randomly initialized and learnable. We ex- pect each query to represent a certain pattern of the brain (refer to visualizations in Section 4.8). The design of keys will be discussed below.
Multimodal fMRI Decoding Beyond text, recent studies have explored decoding brain signals into other modali- ties, including images (Scotti et al., 2024b; Wang et al., 2024a; Chen et al., 2023c), videos (Sun et al., 2024; Chen et al., 2023d), speech (Chen et al., 2024) and music (Denk et al., 2023), as well as in reverse directions (Toneva & Wehbe, 2019). The encoder architectures in these works typ- ically either adopt transformers for time-series data, which distinct from our voxel-based setting, or fall into one of the architectures we have discussed in previous paragraphs. Therefore, focusing on text generation, our work improves the fMRI encoder architecture for voxel-level inputs, and designs novel objectives (i.c., brain instruction tuning) for our target modality.
2024a) or sampling- bansld
Exclude fMRI values from keys The vanilla cross atten- tion (Zhu et al., 2020; Vaswani, 2017) derives both keys and values from the same input source. However, we found this would lead to poor performance in fMRI. We argue the reason: different from images or text, which are usually con- sidered translation- -nntaiiionnnvaaiannt,, the positions of voxels carry specific brain b functional information, as voxels in different areas are associated with distinct brain functions. Conse- quently, a voxel's position alone can theoretically serve as effective keys for attention weight computation (McManus, 2002; Zhang et al., 2023c; Liu et al., 2024). Including IMRI values into keys, however, introduces additional noise instead of valuable information, thus resulting in poorer performance, Moreover, since brain regions tend to serve similar functions across individuals, decoupling voxel posi- tions from fMRI signals can facilitate the sharing of priors across subjects, potentially improving generalization to un- seen subjects.
3. Method
In this section, we propose cuuroscienne-inffmmen fMRI encoder designed to achieve high-performance, subject- agnostic decoding. To further enable versatile decoding, we introduce the construction of a brain instruction tuning dataset, which captures diverse semantic representations encoded in fMRI data.
3.1. Method Overview
As illustrated in Figure 3, our model consists of an fMRI encoder fo and an off-the-shelf LLM. In practice, we use
In light of this, instead of the vanilla cross attention, which derives the keys and values from the same inputs, we ex-
3

---

MindLLM: A Subject-Agnostic and Versatile Model for MMRI-to-Texl Decoding
clude the fMRI value of each voxel and use its positional information alone as its key embedding. The positional in- formation is encoded from the coordinates of each voxel, i.e.
Responso
Tokons
Language Model
k = PE(c,) for the i-th voxel, where E R denotes the coordinates of the voxel. In practice, we use the Fourier positional encoding proposed in (Tancik et al., 2020) due to its superiority in encoding coordinate information.
Tokons
Neuroscience- Attention
Idei Encoder
Allerton Lager
Incorporation of Brain Parcellations While positional encoding alone improves performance, it lacks inherent neu- roscientific grounding, potentially making it challenging for the model to efficiently learn representations aligned with es- tablished principles of brain function. To overcome this, we incorporate existing brain region parcellations (Glasser et al., 2016; Rolls et al., 2020) into the key embeddings. Formally, given a parcellation P. with regions indexed by 1, .Np. Let P(t) E 21.2.... Npp be the region that the i-th voxel belongs to, and [i) E Rd be the corresponding learn- able embedding of the region, which will be incorporated in the key embeddings as k. = E[P(i)] E Id.
V
K
(f. +)
(Foctional Eno.)
Fiie-iioale
Query Embbeddigg ngs. [Laaamato s)
Vaoel
Fri-viuul
HCP MMP!
(Rogion Enc.)
Gownoe-Sooe
Figure 3. Model Architecture. The fMRI encoder maps fMRI to a series of fMRI tokens through our proposed heuroscience-infoomeed attention. The large language model, with both fMRI and text tokens, will be trained by brain instruction tuning.
guage tokens in conversations.
Combining Multiple Parcellations It is crucial to choose an appropriate brain region parcellation. Previous region- based methods (Qiu et al., 2023; Li et al., 2021; Kan et al., 2022) can usually only choose one arbitrarily. In contrast, our model design allows us to combine multiple parcella- tions Pl, pp? ... by concatenating their respective region encodings to the key embeddings. In conclusion, the fi- nal key embeddings are the concatenation of the positional encoding and multiple region encodings,
In addition, we provide a side-by-side comparison between our encoder and existing ones in Appendix F.
3.3. Brain Instruction Tuning (BIT)
To enable versatile IMRI-to-text decoding, an appropriate BIT dataset required, yet no such dataset currently ex- ists. To bridge this gap, construct one based on the fact: MSCOCO images (Chen et al., 2015) serve as stimuli for fMRI recordings in the fMRI study (Allen et al., 2022), and an abundance of datasets provide text annotations (e.g.. VQA) for MSCOCO images. Using the images as interme- diaries, we select those relevant to brain functions and pair the fMRI data with corresponding text annotations. For ex- ample, given image of billboard with annotated textual content, we can reasonably infer that when a subject per- ceives textual information (e.g., contents on the billboard), corresponding representations are encoded in the brain. This suggests the possibility of extracting such information from fMRI signals. We select datasets to fulfill various purposes, enabling the model to capture diverse aspects of semantic information embedded in fMRI signals, including visual perception & scene understanding, language & symbolic processing, memory & knowledge retrieval and complex reasoning, which are considered among most fundamental and essential properties of human brains (Robertson, 2002; Stenning & Van Lambalgen, 2012; Wade & Swanston, 2013; Friederici, 2017).
k; 1 |
rog.
(1)
where| || denotes the concatenation operation. This process is illustrated in Figure 3's lower right part.
The positional and region encodings complement each other: The region encodings serve as coarse-scale features, provid- ing neuroscientific- grounded basis, while the fine-scale positional encoding allows our model to learn finer-grained information directly from the data.
This attention design separates a voxel's functional informa- tinh-which is largely consistent across individuals-from its IMRI value, thereby enhancing generalization. Instead of relying on pooling or sampling, the attention mechanism employs learnable aggregation, while the integration of po- sitional encoding and neuroscientifically informed region encodings further ensures high performance.
After the attention layer, we obtain the hidden representa- tions 2y E N. where Ng is the number of query embed- dings. We then employ an MLP and a reshape operation to map the hidden representations to L. fMRI tokens, i.e.,
Perception & Scene Understanding As illustrated in Fig- ure 4, we begin by using caption tasks at both coarse and fine-grained levels to train the model's ability to understand and summarize what the subject perceives visually (Chen et al., 2015; Krause et al., 2017). Additionally, we incor- porate QA tasks (Ren et al., 2015; Krishna et al., 2017:
X, = reshape (MLP(1z,)) E R Lxd.
The process of the fMRI encoder is illustrated in Figure 3. The obtained fMRI tokens are then prepended to the lan-
4

---

MindLLM: A Subject-Agnostic and Versatile Model for IMRI-to-Texi Decoding
sample v, we sample multi-run conversation X, X²)) from all conversations associated with it, where T 2 represents the number of turns. indicates the message from the assistant and u indicates the message is from the user. The training objective is to maximize the probability of the assistant's response only
Preoept Scate Undarslanding
Haspuga 5000 DA UA Tenyca
(X X a: X
Martory Know odge
Procisss Caplion A DIVVA
Brain Intruction Turing
Longuege & Byrbboli Picceesing
STVDAA DRIGN A-R0/A T5ay4
Complex Reasoning
Lizaa intruciion 193K, UMG manuete VONE
Figure Dataset Taxonomy in Brain Instruction Tuning.
T II r=1
arg max p( XT. Xinst)
x X Xisss, X,)
Acharya et al., 2019) to enhance the model's ability to re- trieve and reason about visually perceived content.
Figure 5 illustrates the chat template and the training ob- jective. We freeze the weights of the LLM and only train the fMRI encoder since we want to preserve the LLM's language modeling prior and ensure a fair comparison with baselines such as Xia et al. (2024).
Memory & Knowledge Retrieval To go beyond tasks di- rectly related to present visual perception, we construct the previous captioning task, a mmemory- task that challenges the model to caption images that the subject pre- viously viewed, simulating memory recall processes. Fur- thermore, we aim to encode knowledge structures in human brains. The OK-VQA (Marino et al., 2019) and A-OKVQA (Schwenk et al., 2022) datasets include questions requir- ing external knowledge that is not present in the image but resides in human brains. For example, A photo of hy- drant may prompt the answer "firetruck," even though the firetruck is absent in the image. This association also reflects the way human cognition operates through a network of in- terconnected meanings, where one concept unconsciously triggers another. Such process, which is called "slippage of the signifier" (Lacan, 2001; 1988; Miller & Lacan, 2018), highlights the symbolic processes through which the brain constructs and retrieves meaning.
Computational Complexity According to the analysis in Appendix C. our model does not introduce additional com- plexity compared to previous methods (Scotti et al., 2024b; Wang et al., 2024a).
<system message> user: X.,, Xinat- XI"
assistant: Xf
user: XI
assistant: X
Figure 5. The chat template used during instruction tuning, illus- trating two turns of conversations. Two turns of conversation are shown. Tokens highlighted magenta are used for next-token prediction loss computation.
Language & Symbolic Processing In addition to the aforementioned OK-VQA and A-OKVQA datasets, which are also related to symbolic process, we further combine datasets of text recognition (Biten et al., 2019) and numeri- cal reasoning (Acharya et al., 2019) to facilitate this aspect.
4. Experiments
In this section, we first evaluate our model on various down- stream tasks, demonstrating its versatile decoding capa- bilities. Next, assess its generalizability to novel sub- jects and its adaptability to real-world applications. Finally, we analyze the functions of queries in our neuroscience- informed attention mechanism.
Complex Reasoning Finally, we try to approximate the rea- soning process that happens in human brains with datascts (Liu et al., 2023; Wang et al., 2023a; Li et al., 2018) that require intricate logical and inferential processes. We expect these datasets challenge the model extract the reason- ing process, drawing upon both visual understanding and abstract mroblem-solvingg thus bridging perception, memory, and knowledge into a cohesive cognitive framework.
4.1. Settings
[MRI Datasets We use the widely used Natural Scenes Dataset (NSD) (Allen et al., 2022), a large-scale dataset consisting of fMRI measurements of 8 healthy adult subjects. During data collection, subjects viewed images from the MS-COCO dataset (Lin al., 2014) and were instructed to press buttons to indicate whether they had previously seen each image.
We ended up with a brain instruction tuning dataset consist- ing of 980, 610 conversations associated with fMRI record- ings from 15 datasets. Appendix A lists the instructions and other details for each dataset. The instruction tuning enables versatile fMRI-to-text decoding. In particular, the introduc- tion of tasks like previous caption empowers the model to perform a broader range of tasks beyond vision-related ones, which the previous model (Xia et al., 2024) fails.
Downstream Datasets The downstream dataset will be discussed within each experiment section. See examples and a short description for all datasets we will use in Appendix A. Implementation details could be found in Appendix B.
To train the model with the BIT dataset, for each
5

---

MindLLM: A Subject-Agnostic and Versatile Model for IMRI-to-Texi Decoding
4.2. Brain Captioning
the versatility of the model, i.e., its performance on various downstream tasks.
To evaluate the model's performance on downstream tasks, start with the widely used brain captioning benchmark (Xia et al., 2024). The task, built upon COCO Caption (Chen et al., 2015) requires the model to predict captions of given images as IMRI stimuli.
Baselines Besides baselines that could be adapted to this experiment from the previous one, we further consider the following subject-agnostic models as baselines. 1) Mind- Bridge (Wang et al., 2024a) flattens the voxels and adap- tively adjusts the padding and stride to pool the voxels into fixed dimension. The original implementation of Mind- Bridge has subject-specific parameters. However, since those parameters are of the same size, we make them shared across subjects and thus make the model subject-agnostic 2) UniBrain (Wang et al., 2024b) samples voxels into a fixed number of groups and employs a transformer where groups are treated as tokens. This UniBrain is unrelated to the UniBrain in the previous section; they just share the same name.
Baselines The following baselines are considered in this ex- periment: SDRecon (Takagi & Nishimoto, 2023), UniBrain (Mai & Zhang, 2023), and BrainCap (Ferrante et al., 2023) employs linear regression, mapping the fMRT to the inputs of an image caption model (Li et al., 2023). OneLLM (Han et al., 2024a) is multimodal large language models that align 8 modalities (including fMRI) with language all one model. For fair and efficient comparison, we only finetune the encoder, given that we freeze the LLM in our method as well. UMBRAE learns an encoder that maps fMRIs to images through encoder similar the MLP mixer (Tol- stikhin et al., 2021). BrainChat (Huang, 2024) segments the flattened voxels into 16 patches and employs transformer to decode text conditioned on the patches. It is worth noting that all of these baselines require subject-specific layers or parameters. contrast, our model subject-agnostic, thus with the potential to generalize on novel subjects.
Datasets & Metric We use the test split of all QA & caption datasets in the BIT dataset. We strictly adhere to the official metrics on all datasets. In summary, for sentence generation, we use BLEU-k (Papineni et al., 2002), ROUGE-L (Lin, 2004), CIDEr (Vedantam et al., 2015), SPICE (Anderson et al., 2016), METEOR (Banerjee & Lavie, 2005). For QA- related tasks, well special metrics proposed in the original paper (e.g. ANLS for ST-VQA (Biten et al., 2019)).
VQA accuracy (Antol et al., 2015) as
five standard met- rics for text generation: BLEU-k (Papineni et al., 2002), ROUGE-L (Lin, 2004), CIDEr (Vedantam et al., 2015), SPICE (Anderson et al., 2016), METEOR (Banerjee & Lavie, 2005).
Metric Following previous works,
The results on subject are shown in Table 2. Our model out- performs baselines, with an average improvement of 12.0%. Further, by comparing instruction tuning and from-scratch models, find that instruction tning has significant pos- itive effect, with an average improvement of 28.0%. The results remain stable across different random seeds; for in- stance, according to our observations, the BLEU-1 score for paragraph captioning exhibits a maximum of +0033 variance. We report experiments on other subjects Appendix G.
Table shows that our model outperforms baselines in terms of most metrics, with average improvement of 3.32%, even if our model does not have any subject-speciffe layers. We argue that this is attributed to both the novel architecture design and the introduction of BIT, which will be evident in the next experiment.
4.4. Unseen Subject Generalization
4.3. Versatile Decoding
Our neuroscienee-iforreed, subject-agnostic design en- hances generalization to novel subjects, a crucial factor in real-world applications where training model for each
The purpose of experiments in this section is two-fold: 1) To investigate the impact of our model design and the introduc- tion of BIT on performance improvement. 2) To evaluate
Table 1. Results of brain captioning. The CIDEr metric is scaled by a factor of 100 for consistency with Table I and baselines.
subject agnostic
Method
BLEU-1
BLEU-2
BLEU-3 1
BLEU-4
METEOR
ROUGE
CIDEr 1
SPICE
SDRecon (Takagi Nishimoto, 2023)
36.21
17.11
7.22
10.03
25.13
13.83
5.02
OneLLM (Han et al., 2024a)
47.04
26.97
15.49
9.51
13.55
35.05
22.99
6.26
UniBrain (Mai Zhang, 2023)
16.90
22.20
BrainCap (Ferrante al., 2023)
55.96
36.21
22.70
14.51
16.65
40.69
41.30
9.06
BrainChat (Huang. 2024)
52.30
29.20
17.10
10.70
14.30
45.70
26.10
UMBRAE (Xia ul., 2024)
59.44
40.48
27.66
19.03
19.45
43.71
61.06
12.79
UniBrain (Wang al., 2024b)
59.08
39.64
26.36
17.08
17.49
43.48
48.20
9.38
MindLLM (Ours)
61.75
42.84
29.86
21.24
19.54
45.82
60.97
11.79
6

---

MindLLM: A Subject-Agnostic and Versatile Model for IMRI-to-Texi Decoding
Table Versatile decoding. dash means model could not perform this task. The superscript means the model trained from scratch, in contrast to their BIT version. The CIDEr metric is scaled by factor of 100 for consistency with Table and baselines.
OneLLM
UMBRAE
BrainChat
MindBridge"
UniBrain"
MindLLM
MindBridge
UniBrain
MindLLM
subj agnostic
COCO-QA
Accuracy!
11.09%
22.23%
39.44%
40.19%
38.38%
42.09%
45.33%
42.00%
48.19%
VG-QA
Accuracy?
8.76%
19.67%
21.00%
21.27%
21.68%
23.53%
VQA-v2
Accuracy 1
33.68%
51.23%
40.02%
43.25%
46.04%
44.13%
47.91%
48.58%
52.14%
A-OKVQA
Accuracy
25.
29.20%
50.44%
43.36%
ST-VQA
ANLS1
3.74%
5.46%
9.58%
7.01%
12.76%
11.64%
8.76%
12.92%
OK-VQA
Accuracy
22.98%
10.35%
17.22%
27.63%
18.63%
27.70%
32.13%
32.30%
33.33 %
Accuracy
8.34%
44.10%
43.49%
43.75%
49.46/
53.77%
TallyQA
RMSE
7.45
3.94
1.90
2.03
1.83
2.04
1.86
1.67
1.76
BLEU-1*
0.26
29.82
22.21
21.82
25.69
26.49
25.69
28.28
29.43
BLBU--2
0.08
14.26
10.23
10.47
12.62
12.48
13.00
15.47
15.78
BLEU-3
0.03
6.52
6.38
5.58
6.70
6.43
7.10
8,90
9.14
Paragraph Caption
BLEU-4*
0.01
2.95
2.12
3.11
3.81
3.63
1.22
5.60
5.51
METEOR
2.36
12.60
9.10
10.95
11.13
10.71
11.39
13.50
13.18
0.00
6.02
7:50
3.55
1.82
Accuracy
47.84%
44.55%
48.48%
48.39%
BLEU-1
35.99
35.63
35.30
35.06
36.18
37.26
37.70
BLEU-2
7.44
14.76
18.33
18.27
18.04
17.82
19.38
20.41
20.56
BLEU-31
3.62
8.17
10.01
10.32
10.20
10.05
11.30
12.25
12.34
VQA-E
BLEU-4
1.82
4.87
6.60
6.27
6.14
6.00
7.00
7.83
7.92
CIDEr
19.32
63.26
78.33
79.05
77.31
76.80
86.62
92.08
93.60
METEOR
6.89
12.25
13.64
14.13
13.96
14.81
15.51
15.62
ROUGE
16.84
28.38
32.82
33.78
33.11
31.56
35.87
VQA Acc.
31.41%
10.67%
36.30%
42.00%
37.05%
12.53%
15.95%
11.58%
48.03
ESVQA Acc.
21.02%
0.00%
30.22%
37.40%
32.30%
38.5018
40.97%
37.87%
43.00%
BLEU-1
37.42
23.11
83.99
85.68
83.84
85.88
86.52
85.10
87.10
BLEU-21
31.72
5.86
78.50
81.27
78.51
81.62
82.28
80.01
83.03
FSVQA
BLEU-3
26.95
2.10
73.00
77.10
T3.97
77.62
78.34
75.49
79.27
72.89
73.56
74.35
70.73
METEOR
26.35
41.76
47.59
45.94
47.96
48.63
16.89
49.05
CIDEr
312.75
4.07
600.00
636.40
609.00
646.26
657.02
628.83
666.26
BLEU-1
41.86
21.19
21.17
24.84
44.52
42.45
43.01
47.20
BLEU-2
19.41
8.00
7.57
9.70
22.16
20.01
20.03
25.16
BLEU-3
9.25
1.98
2,85
3.40
10.39
9.61
9.19
12.95
BLEU-4
3.67
1.02
1.28
1.46
5.45
5.31
4.58
7.49
Previous Caption
METEOR
10.14
6.55
7.20
11.00
10.83
10.81
11.96
ROUGE
21.23
20.88
23.01
33.20
32.38
31.99
34.58
CIDEr
6.65
9.21
8.83
11.73
9.39
7.89
7.53
16.02
SPICE
2.19
2.41
2.56
2.78
3.07
2,80
2.92
3.93
individual is impractical. To evaluate it, we perform instruc- tion tuning on 7 out of the 8 subjects the natural scene dataset (Allen et al., 2022), and evaluate generalization on the held-out subject. Table shows our model outperforms two other subject-agnostic baselines in most cases, with average improvement of 24.5% compared to the second- best model. We anticipate further performance gains through the application of domain adaptation techniques (Xiao et al., 2023; Ganin & Lempitsky, 2015; Ganin et al., 2016; Gong et al., 2012), which we leave as a direction for future work.
Dataset & Metrics We use TDIUC (Kafle & Kanan, 2017), QA dataset consisting of 12 types questions, as a bench- mark to evaluate the model's ' various capabilities comprehen- sively. Additionally, we further select task types-sentiment understanding and utility/affrrdanc tasks, that are partic- ularly relevant BCI applications as sub-datasets. The millit/affordanc task, for instance, enables the model to identify useful objects in a given scene and autonomously decide whether to utilize them. Following their paper, we compute the accuracy each type and report the arithmetic mean-per-type (A-MPT) and the harmonic mean per-type (H-MPT). For the selected types, we report the accuracy respectively.
4.5. Adapting to New Tasks
It is common that users want to adapt the MindLLM to their own specific use cases. To this end, we aim to assess our model's s adaptability to new tasks.
Table 4 shows model achieves balanced (high harmonic mean) and consistently improved performances with an

---

MindLLM: A Subject-Agnostic and Versatile Model for IMRI-to-Texi Decoding
Table Model generalization, compared with subject-agnostee model. We train the models on subject 1 - 7 and evaluate on subject which is the held-out subject.
Table 4. Model adaptation to new tasks. sentiment understand- ing and utility/affrrdance are sub-datasets from TDIUC that are particularly relevant to BCI applications.
MindBrridge
UniBrain
MindLLM
COCO-QA
Accuracy f
35.88
24.95
38.96
Method
Overall
Sentiment Understanding
Unilty/Affordanc
VG-QA
Accuracy 1
20.56
16.23
19.29
A-MPT
H-MPT
Accuracy
Accuracy
MindL.IM
41.00%
19.38%
70.00%
0.00%
VQA-v2
Accuracy 1
42.80
40.16
46.04
MindBridge
49.77%
39.88%
80.00%
A-OKVQA
Accuracy f
44.55
28.71
45.54
UniBrain
51.00%
36.70%
80.00%
25.37%
ST:VQA
ANLS
9.33
9.30
13.63
MindLLM
54.08% % 15.43%
80.77%
50.00%
OK-VQA
Accuracy 1
21.94
17.09
25.44
Accuracy 1
38.92
32.51
42.29
Table S. Performance scaling with number of subjects (in-
TallyQA
RMSE
2.12
2.11
distribution). Evaluation is performed on Subject 1, who is in-
BLEU-11
39.81
11.90
44.60
cluded during training.
BI EU-2
19:55
19.67
24.00
BLEU-3
9.29
8.89
12.79
COCO-Caption
METBOR
BLEU-41
10:39
5.21
10.80
4.33
11.16
7.52
sabiccs
sember
111 B1-11
MITTER
RODIIIT
(TTEE
SPICT.
ROUGE1
31.10
31.51
33.47
66.15
57.56
24.40
18.14
18.52
42,113
43:4
8.81
CIDE& 1
8.70
6,40
13.22
59.81
37 16
91.18
16.44
00.51
0.17
2.67
2.39
58:17
38.65
15:49
17:30
16,97
47:30
-9.51
8:10
56.32
30.30
25.95
17.51
17:12
03.87
07:45
0.08
ALEU-11
23.18
21.73
BLEU-2
10.71
8.94
12.64
BLEU-S 31
4.61
3.72
6.27
Puragraph Caption
BLEU-41
2,22
1.92
3.46
Table 6. Performance scaling with number of subjects (out-of- distribution), Evaluation is performed on Subject 8, who is held out during training.
METEOR
9.99
11.18
CIDEr
0.71
1.56
5.61
Accuracy f
41.78
38,53
46.19
BLEU-11 1
32.54
32.86
34.54
BLEU-21
16.13
15.48
17.54
BLBU-1
181.822-4
MITTERR
SPICT:
BLEU-31
8.82
7,98
VQA-E E
1.70
BLEU-4
5.16
4.42
5.81
43.15
10.57
2.50
CIDEr
68.13
58.79
73.27
13.00
10.55
32:53
66:2
2.70
METBOR
12.74
12.26
13.56
07.30
25.55
11.00
31.51
3.1
ROUGE
30.63
29.38
32.36
VQA Acc.
42.33
37.92
43.92
ESVQA Acc. 1
37.16
30,83
39.07
ure 6. The results strongly validate our design. The vanilla cross attention (Pos Enc. Enc.+/MRI) leads to poor performance, while removing fMRI values from the key embeddings (Pos Enc.) yields a significant improvement. Replacing posi- tional encoding with region encodings (Reg. Enc.) accel- erates convergence in the early stages since is grounded by neuroscientific principles. However, it is eventually out- performed by Pos Enc. due to the lack of finerggraieed information. Combining the positional encoding and region encodings (Pos Enc. BReg Enc.) achieves the best model design. In addition, replacing positional encoding with an MLP that maps coordinates to embeddings results in poor performance ((x,y.)+MI.P)) which indicates the amount of high-frequency spatial information in fMRI signals. Addi- tional ablation results evaluation metries are provided in Appendix E.
BLEU-1 1
75.81
82.94
86.19
ESVQA
BLEU-21
71.03
77.24
BLEU-31
66.40
71.91
77.77
BLEU-4
61.55
66.54
73.58
METBOR
45.81
15.21
17.91
CIDER
428.33
587.78
646.30
average of 13.5%. We could also observe the performance benefits from BIT, with 25.0% absolute improvement.
4.6. Performance scale with the number of subjects
We conducted experiments to evaluate how model perfor- mance scales with the number of subjects and report perfor- mances on the COCO caption task. We examined both the in-distribution (seen subjects) setting in Figure 5 and the out- of-distribution (held-out subjects) setting in Figure 6. Our results show significant performance improvements as the number of training subjects increases, demonstrating that the model benefits from exposure to more subjects during pre-training.
4.8. Visualizations and Interpretations
Unlike previous deep learning models (Scotti et al., 2024b; Mai & Zhang, 2023), our model allows interpretations by in- vestigating how queries work in the neuroscience-infomee attention layer. We inspect the attention weights between queries and voxels in Figure 7.
4.7. Ablation Study
We conduct ablation studies on the design of key embed- dings in the neuroscience-iffommer attention module in Fig-
We found that some queries primarily focus on processing
8

---

MindLLM: A Subject-Agnostic and Versatile Model for IMRI-to-Texl Decoding
(Tunik et al., 2007; Connolly et al., 2016). Additionally, queries are also found responsible for communications be- tween high-level brain regions (Figure 7c.7f). Together, these findings indicate that the learnable queries may reflect the dynamics of human brain activities in the visual task, from seeing and thinking about the image to pressing the button for the visual recall task NSD (Allen et al., 2022).
Pos Enc. + (MRI
Pos Enc.
2,00
Valiation Lo3s
1.95
Pos Enc. + Reg Enc.
(c. MLP
1.90
2.85
1.80
1.75
1.70
We provide attention map visualizations of subjects other than subject in Appendix I. We also visualize latent em- beddings during the model forward pass in Appendix H. Qualitative analysis of model responses could be found in Appendix D.
1.0
200
Esach
Figure 6. Ablation study of the key embedding design. Pos Enc. stands for positional encoding. Reg Enc. stands for multiple region encodings. y. MLP means employ MLP map the coordinates to the embeddings instead of positional encoding.
5. Conclusion
In this work, we propose MindLLM, a subject-agnostic and versatile fMRI-to-text decoding model. Our neuroscience- informed attention mechanism in the fMRI encoder ensures subject generalization, while brain instruction tuning en- hances versatility. Comprehensive benchmarks demonstrate our model's state-of-the-an performance, and visualizations offer insights into the patterns leverages. We envision that these advancements will contribute to medical treatments, neuroscience research, and brain tramutcomputer interfaces in the future. Limitations: This work focuses on static fMRI, without incorporating temporal dynamics. Future research could explore integrating temporal information and investi- gating its relationship with other modalities such as videos.
R OBA
(b)
IPS
FartyVis
VislyVis
EDA
SORA
PDM
PBA
(d)
(e)
(1)
Superior
Left Hemisphere
R: Right Hemisphere
Anterie
sterroor
A Acknowledgments
Figure 7. Each subligure corresponds to a specific query of subject 1. We visualize the attention map between that query token and all voxels. Here we randomly select 6 query tokens, each of which exhibits a distinct spatial focus.
We are grateful for funding and computing resource support from Amazon Research, Snap Research, National Science Foundation (NSF) IIS Div Of Information & Intelligent Sys- tems 2403317, Yale AI Engineering Research Seed Grants, Yale Office of the Provost, and Yale Wu Tsai Institute.
single brain regions, such as Parahippocampal Place Area (PPA) (Figure 7a) and Fusiform Face Area (FFA) (7b). As previous research has shown, PPA is related to conceptual association, semantic processing and environmental mem- ory (Eipstein al., 1999; Köhler al., 2002; Bar et al., 2008; Epstein & Ward, 2010) and FFA is known for its critical role in expertise recognition, social cognition and identity mem- ory (Schultz et al., 2003; Tsantani et al.. 2021; Xu, 2005). Both are important brain regions for the conceptualization of visual information and are responsible for the interaction between real-time stimulus and past memory (Brewer et al.,
Impact Statement
This paper presents work whose goal is to advance the field of Machine Learning. There are many potential societal consequences of our work, none of which we feel must be specifically highlighted here.
References
Acharya, M., Kafle, K., and Kanan, C. Tallyqa: Answering complex counting questions. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pp. 8076- 8084, 2019.
1998; Ranganath et al., 2004; Golarai al., 2007).
Moreover, there are some queries that attend to multiple brain regions, revealing the information transmission be- tween low- and high-level brain regions. For instance, inter- actions between early visual areas and higher-level regions like PPA and IntraParietal Sulcus (IPS) (Figure 7e), reveal- ing a potential pattern for human attention- -uiiedd actions
Alayrac J.-B., Donahue, J., Luc, P., Miech, A., Barr, I., Hasson, Y., Lene, K., Mensch, A., Millican, K., Reynolds, M., et Flamingo: a visual language model for few-

---

MindLLM: A Subject-Agnostic and Versatile Model for IMRI-to-Texl Decoding
shot learning. Advances in neural information processing systems, 15:23716-2113 23736, 2022.
Chen, K., Zhang, Z., Zeng, W., Zhang, R., Zhu, E, and
Zhao, R. Shikra: Unleashing multimodal Ilm's referential dialogue magic. arXiv preprint arXiv:2306.11 15195, 2023b.
Allen, E. J., St-Y Yee, G., Wu, Y., Breedlove, J. L., Prince, J. S., Dowdle, L. T., Nau, M., Caron, B., Pestilli, F,, Charest, I., et al. A massive 7t fMRI dataset to bridge cognitive neuroscience and artificial intelligence. Nature neuroscience, 25(1):116-1266 2022.
Chen, X., Fang, H., Lin, T.-Y., Vedantam, R., Gupta, S., Dollár, P., and Zitnick, C. L. Microsoft coco captions: Data collection and evaluation server. arXiv preprint arXiv: 50400035, 2015.
Anderson, P., Fernando, B., Johnson, M., and Gould, S. Spice: Semantic propositional image caption evaluation. In Computer Vision-ECCV 2016: 14th European Confer- ence, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part 14, pp. 382-398. Springer, 2016.
Chen, X., Du, C., Liu, C.,, Wang, Y., and He, H. Open- vocabulary auditory neural decoding using fmriiprompted llm. arXiv preprint urXiv:2405.07840 2024.
Chen, Z., Qing, J., Xiang, T., Yue, W. L.,, and Zhou, J. H. Sceing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding, 2023c. URL https: / /awxiv.vvvvv////////...///aaavva orq/ / abs/ 2211/ 06956.
Antol, S., Agrawal, A., Lu, J., Mitchell, M., Batra, D., Zitnick, C. L.. and Parikh, D. Vqa: Visual question answering. In Proceedings of the IEEE international conference on computer vision, pp. 2425-2433, 2015.
Chen, Z., Qing, J., and Zhou, J. H. Cinematic mind- scapes: High-quality video reconstruction from brain activity, 2023d. URL https: http:::///xvvv...//../ 2305.11674
Banerjee, S. and Lavie, A. Meteor: An automatic metric for mt evaluation with improved correlation with human judgments. In Proceedings of the acl workshop on in- trinsic and extrinsic evaluation measures for machine translation and/or summarization, pp. 65-72, 2005.
Cheng, Z., LLng, S., Zhang, H., Xin, Y., Li, X., Chen, G., Zhu, Y., Zhang, W., Luo, Z., Zhao, D., al. Videollama 2: Advancing spatial-temporal modeling and audio under- standing in video-lims. arXiv preprint arXiv:2406.07476 2024.
Bar, M., Aminoff, E., and Schacter, D. L. Scenes unseen: the parahippocampal cortex intrinsically subserves con- textual associations, not scenes or places per se. Journal of Neuroscience, (8(34)85339-8444 2008
Connolly, J. D., Kentridge, R. W., and Cavina-Pratesi, C. Coding of attention across the human intraparietal sulcus. Experimental brain research, 234:917-930, 2016.
Bernal, S. L., Pérez, M. Q., Beltrán, E. T. M., Pérez, G. M., and Celdrán, A. H. When brain-computer interfaces meet the metaverse: Landscape, demonstrator, trends, chal- lenges, and concerns. arXiv preprint 1rXiv:22/2.03169, 2022.
Denk, T. I., Takagi, Y., Matsuyama, T., Agostinelli, A.,
Nakai, T., Frank, C., and Nishimoto, S. Brain2music: Reconstructing music from human brain activity, 2023. URL https:tpp///... www.aiiivv......om .rgg hatg////////// 221.2.30..1111 11078.
Biten, A. F., Tito, R., Mafla, A., Gomez, L., Rusinol, M., Valveny, E., Jawahar, C., and Karatzas, D. Scene text vi- sual question answering. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 4291- 4301, 2019.
Devlin, J. Bert: Pre-training of deep bidirectional trans-
formers for language understanding. arXiv preprint 2018.
Du, B., Cheng, X., Duan, Y., and Ning, H. fMRI brain decoding and its applications in brain-comman computer interface: A survey. Brain Sciences, 12(2):228, 2022.
Brewer, J. B., Zhao, Z., Desmond, J. E., Glover, G. H., and Gabrieli, J. D. Making memories: brain activity that predicts how well visual experience will be remembered. Science, 281(53800: 1185-1171 1998.
Epstein, R., Harris, A., Stanley, D., and Kanwisher, N. The parahippocampal place area: recognition, navigation, or encoding? Neuron, 23(1) 11)):115-1225, 1999.
Card, N. S., Wairagkar, M., Iacobacci, C., Hou, X., Singer- Clark, T., Willett, F. R., Kunz, E. M., Fan, C., Vahdati Nia, M., Deo, D. R., al. An accurate and rapidly calibrat- ing speech neuroprosthesis. New England Journal of Medicine, 391(7):609-618, 2024.
Epstein, R. A. and Ward, E. J. How reliable are visual con- text effects in the parahippocampal place area? Cerebral Cortex, 20(2):294-303, 2010.
Ferrante, M., Ozcelik, F., Boccato, T., VanRullen, R., and Toschi, N. Brain captioning: Decoding human brain activ- ity into images and text. arXiv preprint arXiv:2305.11560 2023.
Chen, J., Qi, Y., Wang, Y., and Pan, G. Mindgpt: Inter- preting what you see with non invasivvee brain recordings. arXiv preprint 0rXiv:2309.1577 2023a.
10

---

MindLLM: A Subject-Agnostic and Versatile Model for IMRI-to-Text Decoding
Friederici, A. D. Language in our brain: The origins of uniquely human capacity. MIT Press, 2017.
Kafle, K. and Kanan, C. An analysis of visual question answering algorithms. In ICCV, 2017.
Ganin, Y. and Lempitsky, V. Unsupervised domain adapta- tion by backpropagation. In International conference on machine learning. pp. 1180-1 11899119.,..11111111111119911991 PMLR, 2015.
Kan, X., Dai, W., Cui, H., Zhang, Z.,, Guo, Y., and Yang. C. Brain network transformer. Advances in Neural Informa- tion Processing Systems, 35:25586-255809 2022.
Ganin, Y., Ustinova, E., Ajakan, H., Germain, P., Larochelle, H.. Laviolette, F., March, M., and Lempitsky, V. Domain- adversarial training of neural networks. Journal of ma- chine learning research, 17(59):1-35, 2016.
Köhler, S., Crane, J., and Milner, B. Differential contribu- tions of the parahippocampal place area and the anterior hippocampus to human memory for scenes. Hippocam- pus, 12(6):718-723, 2002.
Glasser, M. F,, Coalson, T. S., Robinson, E. C., Hacker, C. D., Harwell, J., Yacoub, E., Ugurbil, K., Andersson, J., Beckmann, C. F., Jenkinson, M., et al. multi-modal parcellation of human cerebral cortex. Nature, 536(7615);
Kondratyuk, D., Yu, L., Gu, X., Lezama, J., Huang, J., Schindler, G., Hornung, R., Bírodkar, V., Yan, J., Chiu, M.-C., et al. Videopoet: A large language model for zero- shot video generation. arXiv preprint ariv:2311..4422. 2023.
171-178, 2016.
Golarai, G., Ghahremani, D. G., Whitfield-Gabriel S., Reiss, A., Eberhardt, J. L., Gabrieli, J. D., and Grill- Spector, K. Differential development of high-level visual cortex correlates with category-specife recognition mem- ory. Nature neuroscience, 10(4):512- 1221222-2222222 2007.
Krause, J., Johnson, J., Krishna, R., and Fei-Fei, L. A hierarchical approach for generating descriptive image paragraphs. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 317-325, 2017.
Gong, B., Shi, Y., Sha, F., and Grauman, K. Geodesic flow kernel for unsupervised domain adaptation. In 2012 IEEE conference on computer vision and pattern recognition, pp. 2066-2073 IEEE, 2012.
Krishna, R., Zhu, Y., Groth, O., Johnson, J., Hata, K., Kravitz, J., Chen, S., Kalantidis, Y., Li, I..J., Shamma, D. A., et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. In- ternational journal of computer vision, 123:32-73, 2017.
Goyal, Y., Khot, T., Summers-Star Staye D., Batra, D., and Parikh, D. Making the vqa matter: Elevating the role of image understanding in visual question answer- ing. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 6904 6913, 2017.
Lacan, J. The Seminar of Jacques Lacan. WW Norton & Company, 1988.
Lacan, J. Ecrits: A selection. Routledge, 2001.
Han, J., Gong, K., Zhang, Y., Wang. J., Zhang, K., Lin, D.,
Li, Li, D.,, Savarese, S., and Hoi, S. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023.
Qiao, Y., Gao, P., and Yue, X. Onellm: One framework to align all modalities with language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 26584-26595, 2024a.
Li, Q.. Tao, Q., Joty, S., Cai, J., and Luo, J. Vqa-e: Explain- ing, elaborating, and enhancing your answers for visual questions. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 552-567, 2018.
Han, X., Wang, Y., Zhai, B., You, Q., and Yang. H. Coco is" all" you need for visual instruction fine-tuning. arXiv 2024b.
preprint ar Xxxv:2401.00900
Hc, K., Chen, X., Xie, S., Li, Y., Dollár, P., and Girshick, R. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and patlern recognition, pp. 16000-16009, 2022.
Li, X., Zhou, Y., Dvornek, N., Zhang, M., Gao, S., Zhuang, J., Scheinost, D., Staib, L. H., Ventola, P., and Duncan, J. Braingnn: Interpretable brain graph neural network for fMRI analysis. Medicul Image Analysis, 74: 102233, 2021.
Hmamouche, Y., Chihab, I., Kdouri, L., and Seghrouchni, A. F. F. multimodal Ilm for the non-invasive decoding of spoken text from brain recordings. arXiv preprint MoXiv:2409.19711 2024.
Lin, C.-Y. Rouge: package for automatic evaluation of summaries. In Text summarization branches out, pp. 74 81, 2004.
Huang, W. Brainchat: Decoding semantic information from MRI using vision lasion-l pretrained models. arXiv preprint ar Aovi:22406.000 07554, 2024.
Lin, T.-Y., Maire, M., Belongie, S., Hays, J., Perona, P., Ra- manan, D., Dollár, P., and Zitnick, C. L. Microsoft coco: Common objects in context. In Computer Vision-ECCV

---

MindLLM: A Subject-Agnostic and Versatile Model for IMRI-to-Texi Decoding
2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740- 755. Springer, 2014.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I., al. Language models unsupervised multitask learners. OpenAl blog, 1(8)99 2019.
Liu, C., Jing, J., Jiang, J., Wen, W., Zhu, W., Li, Z.., Pan, Y., Cai, X., Liu, H., Zhou, Y., et al. Relationships be- tween brain crructueeffnctionn coupling in normal aging and cognition: A cross-ethnicity population-banl baased study. Neurolmage, 299:120847. 2024.
Radford, A., Kim, J. W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., al. Learning transferable visual models from natural language supervision. In International conference machine learning, pp. 8748 8763. PMLR, 2021.
Ranganath, C., DeGutis, J., and D' D'Esposito, M. Category- specific modulation of inferior temporal activity during working memory encoding and maintenance. Cognitive 2004.
Liu, F., Emerson, G., and Collier, N. Visual spatial reason- ing. Transactions of the Association for Computational Linguistics, 11:63-6551, 651, 2023.
Brain Research,
Luo, A. F., Henderson, M. M., Tarr, M. J., and Wehbe, L.. Brainscuba: Fine-grained natural language captions of vi- sual cortex selectivity, arXiv preprint arXiv:2310.04420, 2023.
Ren, M., Kiros, R., and Zemel, Exploring models and data for image question answering. Advances in neural information processing systems, 28, 2015.
Robertson, L. T. Memory and the brain. Journal of dental education, 66(1):30-44 42, 2002.
Mai, W. and Zhang, Unibrain: Unify image reconstruc- tion and captioning all in one diffusion model from human brain activity. arXiv preprint arXiv:2308.00 07428, 2023.
Rolls, E. T., Huang, C.-C., Lin, C.-P., Feng, J., and Joliot, M. Automated anatomical labelling atlas 3. Neuroimage, 206: 16189, 2020.
Marino, K., Rastegari, M., Farhadi, A., and Mottaghi, R. Ok-vqa: A visual question answering benchmark requir- ing external knowledge. In Proceedings of the IEEF/vvf conference on computer vision and pattern recognition, pp. 1199-32044 2019.
Schultz, R. T., Grelotti, D. Klin, A., Kleinman, J., Van der Gaag, C., Marois, R., and Skudlarski, P. The role of the fusiform face area in social cognition: implications for the pathobiology of autism. Philosophical Transactions of the Royal Society of London. Series B: Biological Sciences, 358( 1430):4155 427, 2003.
McManus, I. C. Right hand, left hand: The origins of asym- metry in brains, bodies, atoms, and cultures. Harvard University Press, 2002.
Schwenk, D., Khandelwal, A., Clark, C., Marino, K., and Mottaghi, R. A-okvqa: A benchmark for visual ques- tion answering using world knowledge. In European conference on computer vision, pp. 146-162. Springer, 2022.
Miller, J. A. and Lacan. The four fundamental concepts of
Routledge, 2018.
Murahari, V., Chattopadhyay, P., Batra, D., Parikh, D., and Das, A. Improving generative visual dialog by answer- ing diverse questions. arXiv preprint arXiv: 1909. 10470, 2019.
Scotti, P., Banerjee, A., Goode, J., Shabalin, S., Nguyen, A., Dempster, A., Verlinde, N., Yundler, E., Weisberg, D., Norman, K., et al. Reconstructing the mind's eye: fMRI- to-image with contrastive learning and diffusion priors. Advances in Neural Information Processing Systems, 36, 2024a.
Papineni, K., Roukos, S., Ward, T., and Zhu, W.-J Bleu: method for automatic evaluation of machine transla- tion. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics, pp. 311-318, 2002.
Scotti, P. S., Tripathy, M., Villanueva, C. K. T., Kneeland, R., Chen, T., Narang, A., Santhirasegaran, C., Xu, J., Naselaris, T., Norman, K. A., et al. Mindeye2: Shared- subject models enable fMRI-to-mmaee with hour of data. arXiv preprint arXiv:2403. /207, 2024b.
Qi, Z. Dong, R., Zhang, S., Geng, H., Han, C., Ge, Z., Yi, L., and Ma, K. Shapellm: Universal 3d object understanding for embodied interaction. In European Conference on Computer Vision, pp. 214-238. Springer, 2025.
Shin, A., Ushiku, Y., and Harada, T. The color of the cat gray: million full-sentences visual question answering (fsvqa). arXiv preprint arXiv:1609.06657. 2016.
Qiu, W., Chu, H., Wang, S.. Zuo, H., X., Zhao, Y., and Ying, R. Learning high-order relationships of brain re- gions. In Forty-first International Conference on Machine Learning, 2023.
Stenning, K. and Van Lambalgen, M. Human reasoning and cognitive science. MIT Press, 2012.
12

---

MindLLM: A Subject-Agnostic and Versatile Model for IMRI-to-Texl Decoding
Sun, J., Li, M., Chen, Z., and Moens, M.-F. Neurocine: Decoding vivid video sequences from human brain tivties, 2024. URL https://arxivv org/ abs/ 2402. 01590.
Wang, S., Liu, S., Tan, Z., and Wang, X. Mindbridge: A cross-subject brain decoding framework. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 11333-11342, 2024a.
Takagi, Y. and Nishimoto, S. High-resolution image re- construction with latent diffusion models from human brain activity. In Proceedings of the IEEE/CVF Confer- ence on Computer Vision and Pattern Recognition, pp. 14453-14463, 2023.
Wang. W., Lv, Q. Yu, W., Hong, W., Qi, J., Wang, Y., Ji, J., Yang, Z., Zhao, L.., Song, X., et al. Cogvlm: Visual expert for pretrained language models. arXiv preprint arXiv:2311.03077 2023b.
Wang. Z.,
Zhao, Z., Zhou, L.,, and Nachev, Unibrain: unified model for cross-subject brain decoding. arXiv preprint arXiv:2412.19487, 2024b.
Tancik, M., Srinivasan, P., Mildenhall, B., Fridovich-Keil, S., Raghavan, N., Singhal, U., Ramamoorthi, R., Barron, J., and Ng, R. Fourier features let networks learn high fre- quency functions in low dimensional domains. Advances in neural information processing systems, 33:7537-7547, 2020.
Xia, W., de Charette, R., Öztireli, C., and Xue, J.-H. Um- brae: Unified multimodal decoding of brain signals. arXiv preprint arXiv:2404.07200 2024.
Tolstikhin, I. O., Houlsby, N., Kolesnikov, A., Beyer, I.., Zhai, X., Unterthiner, T., Yung, J., Steiner, A., Keysers, D., Uszkoreit, J., et al. Mlp-mixer: An all-mlp architec- ture for vision. Advances in neural information process- ing systems, 0124261-242772 2021.
Xiao, Z., Wang, H., Jin, Y., Feng, L.., Chen, G., Huang, F,, and Zhao, J. Spa: A graph spectral alignment perspective for domain adaptation. Advances in Neural Information Processing Systems, 36:37252-37272 2023.
Xu, R., Wang, X., Wang. T., Chen, Y., Pang. J., and Lin, D. Pointllm: Empowering large language models to un- derstand point clouds. arXiv preprint arXiv:2308. 16911, 2023.
Toneva, M. and Wehbe, L. Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain). Advances in neural information processing systems, 32, 2019.
Xu, Y. Revisiting the role of the fusiform face area in visual expertise. Cerebral Cortex,
2005.
Tsantani, M., Kriegeskorte, N., Storrs, K., Williams, A. I.., McGettigan, C., and Garrido, L. Ffa and ofa encode distinct types of face identity information. Journal of Neuroscience, 1959:1952-1999, 2021.
Yu, J., Wang, Z., Vasudevan, V., Yeung. L., Seyedhosseini, M., and Wu, Y. Coca: Contrastive captioners are image- text foundation models. arXiv preprint arXiv:2205.01911, 2022.
Tunik, E., Rice, N. J., Hamilton, A., and Grafton, S. T. Be- yond grasping: representation of action in human anterior intraparietal sulcus. Neuroimage, 36:777-T86, 2007.
Zhang, H., Li, X., and Bing, L. Video-llama: An instruction- tuned audio-visual language model for video understand- ing. arXiv preprint wxXiv:2306.02888 2023a.
Vaswani, A. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Zhang, P., Dong, X., Wang, B., Cao, Y., Xu, C., Ouyang, L., Zhao, Z., Duan, H., Zhang, S., Ding, S., et al. ditemllm-xcomposet A vision-language large model for advanced text-image comprehension and composition. arXiv preprini arXiv:2309. 15/ 2023b.
Vedantam, R., Lawrence Zimick, C., and Parikh, D. Cider: Consensus-basse image description evaluation. In Pro- ceedings of the IEFE conference computer vision and pattern recognition, pp. 4566-4575, 2015.
Zhang, X., Liang, C., Wang, N., Wang, Y., Gao, Y., Sui,
Wade, N. and Swanston, M. Visual perception: An introduc- tion. Psychology Press, 2013.
C., Xin, H., Feng, M., Guo, L., and Wen, H. Abnormal whole-brain voxelwise structure- sructure-function coupling and its association with cognitive dysfunction in patients with different cerebral small vessel disease burdens. Frontiers Aging Neuroscience, 15:1 148738, 2023c.
Wang, J., Yang, Z., Hu, X., Li, L., Lin, K., Gan, Z., Liu, Z., Liu, C., and Wang, L. Git: A generative image-to- text transformer for vision and language. arXiv preprint arXiv:2205. /4100, 2022.
Zheng, L.,, Chiang, W.-L.,, Sheng, Y., Zhuang, S., Wu, Z., Zhuang, Y., Lin, Z., Li, Z., Li, D., Xing, E., et al. Judging llm-as-a-judge with mt-bench and chatbot arena. Ad- vances in Neural Information Processing Systems, 36: 46595-46623, 2023.
Wang, J., Meng, L., Weng, Z., He, B., Wu, Z., and Jiang, Y.G.. To see is to believe: Prompting gyt-4v for better vi- sual instruction tuning. arXiv preprint arXiv:2311.07574, 2023a.
13

---

MindLLM: A Subject-Agnostic and Versatile Model for IMRI-to-Texi Decoding
Zhu, X., Su, W., Lu, L., Li, B., Wang, and Dai, J. De- formable detr: Deformable transformers for end-to-end object detection. arXiv preprint arXiv:2010.04115,, 2020.
14

---

MindLLM: A Subject-Agnostic and Versatile Model for IMRI-to-Texi Decoding
A. Dataset Details
A.1. Details each dataset brain instruction tuning
In this section, we give a brief description each source of our brain instruction datasets as well as examples from them.
Table 7: Dataset details and examples.
Dataset
Description
Example
Generating one-scutence caption of the image that the subject previ- ously saw.
Previous Caption
A neat bedmom pains modern chairs with glass table.
COCO Caption (Chen ot al., 2015)
Generate one-sentence caption the image the subject currenlly sees.
pedestrian walking down highway the bus.
An clephant with harness, and seen in a ditt field on a sunny day. The seat on the back the elephant sun is shining through the trees onto the ground below. Behind the clephant there cinder block wall with grass growing in front of the wall. There are troes behind the
his back is
Image Paragraph Captioning (Krause et al., 2017)
Generate one purugruph cuption image subject currently sees
brown and tied with rupes. The
COCO QA (Ren al,, 2015)
Answer questions uccording to the image.
what leaves from basket enclosure? [A]: Giraffe.
Answer image- questions require richer semantic understand- ing image than COCO-QA.
Visual Gemome QA (Krishna al., 2017)
Where the photo taken? [A]: office.
VQAv2
Answer image- based questions with better equality and diversity than COCO-QA.
HU
(Goyal et al., 2017)
[Q]:
hu
OK-VQA
Answer image-based questions that roquise external knowledge beyond the image itselt.
(Marino et al., 2019)
[Q]: What part of the body do you wear rightmost objects on! [A]: Neck.
ST-VQA
Answer questions of high-level se- information present as the textual cue
(Biten al. 2019)
[Q] What is written on the front of the shirt? [A]: Marine.
TallyQA
Count objects present images.
(Acharya et al., 2019)
[Q]: How many bats the wall? [A]:
VQA-E (Li dieetll al., 2018)
questions gemerate corresponding explanations for an image-based question.
[Q]: Are the people going for a walk in the forest? [A]: Yos. Here explanation: A picture the land, trees. and people passing by as they ride in vehicle.
A-OKVQA (Schwenk et al., 2022)
Answer multiple-choice questions.
season up next? Multiple Choices: tumn B. spring C. summer D. winter [A]: B
Continued on next page
15

---

MindLLM: A Subject-Agnostic and Versatile Model for IMRI-to-Text Decoding
Dataset
Description
Example
ISVQA
the questions full lences.
[Q]: this area that is more populated with pedestrians than cars? [A]: Yes, an area that is more populated with pedestrians than
(Shin et al,, 2016)
VisDial
answers imaage questions in multi turn dialogue,
[Q]]: Is the photo color? [AI]: No. [Q2]: Is the man wearing glasses [A2]: Can't tell. [Q3]: How many horses are there? IA3]:2
(Murahari et al., 2019)
activity the person AMRI? [AI]: The main activity of the person the fMRI is brushing her teeth. [Q2]: What should considered when muintaining ural hygiene in public setting? [A2]: When maintaining oral hygiene in a public setting (as depicted in the fMRI with green toothbrush while carrying shoulder bag). factors sach eleanliness, privacy, management, personal bygiene supplics should be taken into account...
[QI]:
Geuerate for object-lovel and soene level uns werting or rea- soning questions single round conversations.
Instruction (Liu al., 2023)
brushing her teeth
[Q]] What is the primary object in the IMR1? [A]]: The primary object in the FMRI is hird. [Q2]: Can deseribe the color and appearance of the bird? [A2]: The deep are distinctively surrounded by yellowish huc. The bird also possesses bluc-tinted beak. [Q3]; How many birds the picture?' [A3]: There only bird in the picture...
LVIS InstrucH (Wang et al., 2023u)
Similar to LLava Instruct-IV finer-gren iner-grained and higher quality.
body
darker
eyes
Generate answers Image-hased questions from 12 different task- directed question types such as ob- ject presence and uillity/lllordane.
TDIUC (only used downstream) (Kalle & Kanun, 2017)
[Q]: What ubject can be thrown? [A]: Baseball
A.2. Dataset statistics
Table 8 summarizes the statistics of each subject in the natural scene dataset (Allen et al., 2022).
Table 8. Number of voxels for each subject.
subject
2
3
4
5
6
7
8
#(input voxels)
15724
14278
15226
13153
13039
17907
126682
14386
#(samples)
30000
30000
24000
22500
30000
24000
30000
22500
A.3. Instructions
Table 9 lists instructions for each dataset (i.e. Xina). Some instructions are inspired by (Han et al., 2024b), For question- answering tasks, the instructions are appended after the question in a new line.
B. Implementation details
We choose Vicuna-7b (Zheng et al., 2023) as our backbone LLM. During the brain instruction tuning stage, We use AdamW the optimizer, with the learning rate of 1 x 10-3, weight decay of 0.01 and 3n 0.9, /3 0.999. We do not use machine with 8x L40S GPUs
learning rate scheduler. We set the batch size to 64. The instruction tuning is conducted for 8 days. And each downstream fine-tuning is conducted on single L40S GPU with 1 x 10-1 learning rate and 48 batch size. For generations, we have adopted the greedy decoding strategy.
We use 128 as the number of fMRI tokens. We use 4-layer MLP in the fMRI encoder. We set the number queries to 1024. The dimension of the query embeddings 128.
16

---

MindLLM: A Subject-Agnostic and Versatile Model for IMRI-to-Texi Decoding
Table Instruction template and statistics of the BIT dataset. "-" indicates that the instruction emhedded within the conversation and does not require an additional one. Note that not all conversations are associated with IMRI recordings as only subset of MSCOCO images were used as stimuli in study. Consequently, the number of usable conversations practice will be lower.
Dataset
#cconversations)
Instruction
system prompt
You are a helpful agent that decodes the brain activity a person looking at an image.
Previous Caption
149, 875
Please describe the image the subject saw previously.
COCO Caption
616, 767
Please describe the fMRI as simply as possible.
Image Paragraph Captioning
9, 598
Describe the fMRI in one paragraph.
COCO QA
117, 684
Answer the question with a short phrase.
Visual Genome QA
676, 116
Answer the question with a short phrasc.
VQAv2
6, 5,581,11 110
Answer the question with a short phrase. Answer the question with a short phrase.
OK-VQA
140, 550
ST-VQA
29, 751
Answer the question with a short phrase.
TallyQA
238, 056
Answer the question with a number. Answer with short phrase and provide explanation for your answer.
VQA-E
697, 860
A-OKVQA
18, 201
Answer with the option's 's letter from the given choices and provide explana-
tion for your choicc.
FSVQA
369, 861
Answer the question in a full sentence.
VisDial
125, 351
Llava Instruct 150K
157, 712
LVIS Instruct4V
222, 711
C. Computational Complexity and Runtime Analysis
In the neuroscience-infommed attention, the complexity of the dot product between queries and keys is O(dN N₂). The complexity of the aggregation of values is ONN,)), which is negligible. The MLP maps the hidden representation of dimension Ng to I. x d, therefore its complexity d...N.- Therefore, the complexity of the fMRI encoder O(NN + dLN,)) OdNN, (L + N)) O(dN,.N) given that < N.
Table 10 summarizes the computational complexity and runtime comparison of different models. Despite its superior performance, our model introduces only marginal increase in encoder-side computational cost compared to other baselines. Notably, the majority of inference time and complexity arises from the LLM component, which is shared across all models. This highlights that the design choices in the encoder, while crucial for performance, do not significantly affect runtime.
D. Qualitative Analysis
In this section, we present a qualitative analysis of model COCO Captioning, COCO-QA, and OK-VQA, comparing its performance against MindBridge (Wang al., 2024a) and UniBrain (Wang et al., 2024b). As shown in Figure 8 and 9, our results demonstrate significant improvements in visual understanding across multiple tasks. The model shows strength in the following areas: Static Object Recognition. The model demonstrates superior accuracy in identifying stationary objects. In comparison with baseline models (MindBridge and UniBrain), our approach shows improvement in spatial context understanding. For example, when analyzing aircraft imagery (e.g., (a) of Figure 8), our model correctly identifies "airplane sitting on the runway" while baselines incorrectly interpret the scene as "flying in the sky" or "flying over city". demonstrating better state-space recognition. Action Recognition. Our proposed model exhibits enhanced capability in distinguishing between similar actions. In sports scenarios (e.g.. (f) of Figure 9), model correctly identifies "catch ball" while both baselines incorrectly predict "serve", indicating improved action-state discrimination. Potential of neuroscience application. The demonstrated improvements in object understanding and action recognition suggest the potential for advancing brain- brain-computer interface technology and neural processing research. The model's enhanced capabilities in distinguishing object states and actions could lead to more effective neural prosthetics and improved assistive technologies for individuals with visual or motor impairments.
17

---

MindLLM; A Subject-Agnostic and Versatile Model for IMRI-to-Texi Decoding
Table 10. Computational complexity and runtime of each encoder and the LLM. For the first 4 lines, we only time the encoder part, excluding the LLM. The runtime the LLM forward pass is reported separately in last row.
Module
Forward Pass (ms/sample)
Complexity
Notation Explanation
Approximate Values
d: hidden size
d = 100
UMBRAE
0.0109
O(dN + dN2))
N: number of voxels
N = 15000
Ng: number of query tokens
N₂ = 200
d: hidden size
d 1000
MindBridge
0.0048
((dN)
N: number of voxels
N 15000
d: hidden size
d = 1000
G: number of groups
500
UniBrain
0.0175
O(dGK + +dN?)
K: voxels in each group
K 30
Ng: number of query tokens
N₂ 200
d: hidden size
d = 100
MindLLM
0.0181
O(NN,N)
N: number of voxels
15000
Ng: number of query tokens
N₂ 100
LLM
0.0621
Ground Caption: A white jet with blue and orange detailing sits the runway.
Icelandair irliner is preparing for take off.
An airplane sitting at the end of runway preparing for takeoff. Ours A large airplane sitting top of an airport runway.
MindBridge A large passenger jet flying in the sky. UniBrain A large airplane is flying over city.
UNCLE
Ground Truth Caption: white jet with blue and orange detailing sits on the runway.
Outdoor clock with metal decoration against blue sky.
Closeup a clock with a sign that reads "Unele Bills". Ours A clock tower with clock on it. MindBridge A sign that says "STOP" pole. UniBrain A red and white airplane is parked on the tarmac.
(b)
Ground Truth Caption: giraffe standing on top of lush green field. Giraffe standing in middle of fenced in area looking to the ground.
giraffe standing the shade of tree.
Ours A giraffe standing in the middle of field.
MindBridge A train is traveling down the tracks the city.
UniBrain A zebra standing in the middle of street.
(c)
Figure Qualitative Analysis of COCO Captioning.
18

---

MindLLM: A Subject-Agnostic and Versatile Model for IMRI-to-Tex Decoding
Question What animal
Quesiions what are byying to climb on rock
kingdom does the animal belong to?
Ground Truth. TruthAAnswer Answer: bears
Ground Truth.Answer bear, mammal
Ours bear
MindBridge giraffes
UniBrainz elephants
Ours mammal
MindBridge bird
UnlBrain bird
aa)
(d)
Questions what is the teenage girl swinging softball
Can you guess the place name where the train is shown in this picture?
Question
Ground TruthAnsww bat
Ours bat
Ground Arswer
MindBridge ball
station
Unilrrain ball
Our station
MindBridge new york
(b)
(e)
UniBrain london
CNCLE BILLS
Questions what metal
Question Why are the men bending down Ground to catch ball, catch Ours catch ball MindBridge serve UniBrainz serve
decoration against blue sky Ground Truth Answar clock
Ours clock
MindBridge tower
UniBrain tower
(c)
(f)
Figure 9. Qualitative Analysis of COCO QA (left column) and -K-VQA (right column).
E. Ablation Studies on Downstream Tasks
Table 11 reports the results of the ablation study on each downstream task. The results validate the design of neuroscience- informed attention.
F. Architecture Comparison
We make side-by-side comparison between our model and related baselines in Table 12. Our model is the only that achieves the four properties listed in the table's 's head.
G. Experiments on Other Subjects
We report experiments on subjects 2, 7, 7 in Table 13. We chose these subjects because they are commonly used in previous works. We observe consistent performance gains over the baselines in the majority of scenarios.
H. Latent Space Visualization
To assess how subject-specific and subject-agnostic information evolves through the model, we visualize the latent embed- dings at various stages of the encoder in Figure 10. The original inputs exhibit clear subject-specific clusters. After passing through the neuroscience-inforeee attention layer, the embeddings remain separable by subject, indicating that subject identity still preserved. As the data flows through successive layers of the MLP, the subject-specific patterns become increasingly mixed, ultimately forming subject-agnostic representation. This transformation facilitates generalization held-out subjects in zero-shot manner.
I. Brain Attention Maps of Other Subjects
We visualize the flat brain attention maps (corresponding to Figure 7a) in Figure 11. We observe that the attention maps exhibit highly similar spatial patterns across subjects, indicating that the model captures shared
19

---

MindLLM: A Subject-Agnostic and Versatile Model for IMRI-to-Texl Decoding
Table 1. Ablation study of the key embedding design. Pos Enc. stands positional encoding. Reg Enc. stands for multiple region encodings. (x, y, 2)+MLP means employ an MLP to map the coordinates to the embeddings instead of positional encoding. Note that unlike Table 2, where we fine-tune the model on each task individually, here we do not fine-tune the model on each individual task and report the performances after brain instruction tuning.
(z, 2) MLP
Pos Enc.
fMRI
Pos Enc.
Reg Enc.
Pos Enc. + Reg Enc.
COCO-QA
Accuracy f
40.57
43.14
46.29
43.90
44.76
VG-QA
Accuracy f
20.29
20.39
21.12
20.83
21.12
VQA-v2
Accuracy 1
45.25
46.06
47.30
47.44
47.86
A-OKVQA
Accuracy f
46.02
44.25
45.13
42. 48
46.90
ST-VQA
ANLS 1
13.70
9.70
12.79
10.61
11.64
Accuracy f
44.63
49.77
52.29
48.84
53.01
Tally QA
RMSE
2.15
1.83
1.75
1.96
1.73
BLEU-1
49.39
54.86
37.03
52.53
58,98
BL.U--2
26.30
33.84
37.12
33.00
39.08
BLEU-3
14.06
20.67
23.63
20.69
25.56
BLEU-4
8.19
13.42
15.51
13.93
17.36
COCO-Caption
METFOR
12.31
14.97
16.41
14.81
17.12
ROUGE
35.45
40.17
42
39.62
43.34
CIDEr
18.26
32.15
41.32
34.25
46.17
SPICE f
3.94
7.14
8.56
6.76
9.24
BL.U--1 f
24.13
24.57
25.90
24.74
27.17
BLEU-2
11.88
13.03
12.41
12.61
14.50
BLEU-3
6.12
7.16
6.10
7.20
7.99
Paragraph Caption
BL.UU-4
3.55
4.24
3.22
4.35
4.67
METEOR
11.19
11.78
9.05
12.12
14.43
CIDEr
3.32
2.35
7.46
4.97
4.30
Accuracy t
46.74
48.17
48.60
48.48
48.95
BLBU-1
35.64
36.19
36.46
35.06
36.65
BLEU-2
18.65
19.38
19.53
18.32
19.58
BL-U--3
10.85
11.40
11.62
10.54
11.57
VQA-E
BLBU-4
6.79
7.12
7.29
6.38
7.31
CIDEr
83.18
88.14
89.24
85.34
88.96
METEOR
14,20
14.81
15.08
14.54
15.08
ROUGE
33.81
35.15
35.17
34.28
34.76
Average
Perplexity
6.39
5.82
5.47
5.73
5.38
Method
cross-subject
unified architecture
shared parameters
using all voxels
MindEye (Scotti et al., 2024a)
x
X
X
MindEye2 (Scotti et al., 2024b)
X
XX
UMBRAE (Xia et al., 2024)
X
X
MindBridge (Wang et al., 2024a)
X
UniBrain (Wang al., 2024b)
MindLLM
Table 12. Comparison of our model and baselines. Concepts of properties: cross-subject: the model performs across subjects; unified architecture: the architecture each subject the same (the parameters might different, e.g.. MLPs with the same size but different parameters); shared parameters: The model uses the same parameters for all subjects; using all voxels: the model uses all voxels as inputs, thereby mitigating the risk of information from voxel omission. Refer Section and Figure for details of this property.
20

---

MindLLM: A Subject-Agnostic and Versatile Model for IMRI-to-Tex Decoding
Table 13. Breakdown performances subjects 2, 5, respectively.
subject
subject
subject
UMBRAB
UniBrain
Mindüridge
MindLLM
UMBRAB
UniBrzin
MindBrigge
MindLLM
UMBRAB
UniBrain
MindBridge
MindLLM
CA
26,85
47.14
49.04
51.71
29.86
45.52
50.57
53.14
26.78
40.06
50.19
50.76
VI-QA
Accuracy
10.38
VQA-22
Accuracy
56.13
A-OKVQA
Accuracy
40.51
03.30
44.25
49.56
05.21
45.13
50.44
50.74
00.54
96.00
65.13
ST-VQA
ANLS
6.01
9.34
10.49
10.85
4.31
9.77
10.49
9.63
6.46
8:26
13.48
13.71
OK VQA
Accurasy
20.70
32.82
33.50
35.73
22.98
30.43
34.96
35.04
17.89
34.00
54.54
34.87
TallQAA
Accuracy
RMSE
42.21
66.98
0.20
67.84
1.65
42.00
54.05
1.64
08.71
69.3
1,00
40.95
63.08
56.20
58.41
1.73
BLBE--1
16.73
14.75
16.29
14:37
14.21
14:18
Parggraph Cuption
BLEU-4
RLBE--1
6.44
3,08
5.30
8-42
8.70
5.46
11.00
6.96
6.84
3:28
8:75
0.23
0.53
8:97
10.11
7.00
2.355
6:45
4.52
7.42
8.05
4.84
51.54
6.77
METEOR
CIDE
7.53
6.00
4.08
8.53
Accurasy
47.62
50.00
52:30
54.92
49.06
30.18
54.21
93.75
48.95
49.48
$3.94
$4.668
B
50.05
38.63
37. 48
38. 13
30.55
37.41
36.07
38.40
20.58
36.71
38, 30
38, 38.41
BLBU-21
14.87
10:60
20.
20.77
14.90
19.06
20.00
21.04
14.05
19:78
20.00
20.96
VQA-E
7.15
8.17
7.58
7.99
7,30
CIDE
64.60
89.50
91. 23.115
97.68
68:47
201.1
96.000
99.48
61.84
900.00
38.21
99.96
METEOR
12.07
15.12
15.56
16.02
12.51
15.37
15.88
16.25
12.09
15.02
16.07
16.16
ROUGET
28.49
86
36.52
28.75
36.43
35.24
36.61
36.79
FSVQA Acc.
40.13
43.57
45.21
45.53
30.09
$4.70
48.07
RL-E--1
18.95
86.03
87.
67.84
19.00
85.90
87.70
K7.89
18.05
85-48
97.20
88.05
FSVQA
BLEL-21
9.11
$1.96
53.14
63.94
11.94
81.52
85.25
83.87
10.89
81.00
$3.42
$4.16
METEOR
5.65
48.06
49.26
49.88 19.88
5.67
48.00
49.57
50.10
5.65
07.26
49.74
50.18
CIDE
3.19
647.62
671.26
681.94
3.20
648. 21
673.99
684.12
3.57
643.44
676.50
684.72
originel inputs
ofter attertion layer
ofter 1st layer in MLLP
after 2nd layer in MLP
eftrr 3rd leyer MLP
after 4th layer MLP
Unversion
Dimersiinn
Unversion
Diversionn
Dimersion
Dimenaion1
Dimersion
Dimemsiio
Dimension
Dimnnsinn
subtect
subject
Model Forward Pass
Figure 10. Visualization of latent embeddings across model layers using T-SNE.
21

---

MindLLM: A Subject-Agnostic and Versatile Model for IMRI-to-Text Decoding
correspondences across human brains, which is expected due the overall anatomical and functional similarity among individuals. At the same time, do observe moderate subject-specific variations in the attention maps. These reflect differences in voxel-level functional signals and individual variability in precise voxel locations. The model is able to accommodate these differences through flexible attention mechanisms.
L
R
L
R
L
R
Subject-2
Subject-3
Subject-4
(a)
(b)
(c)
Subject-5
Subject-6
Subject-7
(d)
(e)
Superior
L: Left Hemisphere
Anterior
Anterior
R: Right Hemisphere
0.5
Figure Figure 7a's query token different subjects. shows similar patterns across subjects, suggesting shared brain orgamization, while also capturing some subject-specific variances.
22


Reviews:
[{"review_id": "bi2vWc0exx", "reviewer": "Reviewer_Gvse", "summary": "This paper proposes MindLLM for subject-agnostic and versatile fMRI-to-text decoding. MindLLM consists of an fMRI encoder and an off-the-shelf LLM. The paper evaluates MindLLM on several fMRI-to-text benchmarks.", "questions": "", "limitations": "", "rating": 2, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**C1**\n\nThe paper claims \"a voxel’s position alone can theoretically serve as effective keys for attention weight computation\", but no evidence is provided.\n\n**R1**\n\nWe would like to point out politely that it is not a guess—it is strongly supported by the ablation study (note the blue line) in section 4.6. As pointed out in lines 403-406, left column,\n\n> The vanilla cross attention (Pos Enc. + fMRI) leads to poor performance, while removing fMRI values from the key embeddings (Pos Enc.) yields a significant improvement.\n> \n\nwhich validates the hypothesis of using a voxel’s position alone.\n\n**C2**\n\nFor Tables 1, 2 & 3, It is not clear how much of the improvement is due to the proposed model versus the use of superior LLMs. It is unfair to compare their performances with other state-of-the-art baselines that do not use the same LLM in Table 1.\n\n**R2**\n\nWe argue that the comparison is fair across Table 1-3, as pointed out in lines 140-142\n\n> In practice, we use Vicuna-7b (Zheng et al., 2023) as our LLM to maintain consistency with our baseline (Xia et al., 2024).\n> \n\nSpecifically,\n\n- **Table 1**: The strongest baseline, UMBRAE, uses Shikra-7b, a vision-language model improved based on Vicuna-7b—consistent with our choice of LLM.\n- **Table 2**: Besides UMBRAE, for other competitive baselines (MindBridge, UniBrain), we use the same LLM backbone as ours.\n- **Table 3**: All baselines share the same LLM backbone as our model.\n\nFurthermore, as discussed in Section 4.2, we observe substantial performance gains attributable to the **brain instruction tuning** and the **encoder design**. This supports our claim that the improvements stem from our proposed approach rather than solely from the choice of LLM.\n\n**C3**\n\nThe results for METEOR, CIDEr and SPICE of the proposed method are only slightly superior to or even inferior to the baselines.\n\n**R3**\n\nWhile our model does not significantly outperform the baselines in terms of METEOR, CIDEr, and SPICE, it offers several key advantages that the baselines lack:\n\n- As shown in Table 1, our model is the only one that is subject-agnostic. In other words, it can generalize to unseen subjects in a zero-shot manner. This is particularly valuable in BCI applications, where users often expect the device to work out of the box without requiring subject-specific training data.\n- As noted in lines 82–83 (right column), our model can handle tasks beyond simple image-stimulus associations—such as answering memory-related questions—while the strongest baseline, UMBRAE, cannot.\n\nWe would also like to emphasize that our method demonstrates consistent and broad improvements across a wide range of downstream tasks. For example, beyond brain captioning, our model outperforms all baselines on 33 out of 38 evaluation metrics reported in Table 2. \n\n**C4**\n\nThe ablation study is not sufficiently convincing. There is no explanation of the task being experimented on, and the authors should provide the final results, not just the loss.\n\n**R4**\n\nWe appreciate the reviewer’s feedback and apologize for the lack of clarity regarding the experimental setup. The ablation study is conducted across all tasks during brain instruction tuning on subject 1, and the reported loss is the average loss across these tasks. We will update the details about settings in the revised version.\n\nOur decision to present loss curves was intentional: As pointed out in lines 405–411, left column, the comparison of the convergence speeds of the orange and green lines helps illustrate the impact of both region and positional encodings. (We do not plot metrics during training at each step since generation is time-consuming)\n\nHowever, we agree that final performance metrics are essential for a complete picture. We have now computed these metrics for most downstream tasks, and the results are shown below. The results are consistent with our original claim based on the loss curves. Note that it also validates our response to the reviewer's concern 1.\n\n[Table](https://anonymous.4open.science/api/repo/MindLLM-FAA4/file/rebuttal/ablation.png)\n\nWe will include the final results of all tasks in the revised version.", "claims_and_evidence": "The paper claims \"a voxel’s position alone can theoretically serve as effective keys for attention weight computation\", but no evidence is provided. It seems more like a guess rather than a fact. It is important to verify the rationale of this operation.", "methods_and_evaluation_criteria": "See the above.", "theoretical_claims": "Not applicable.", "experimental_designs_or_analyses": "- It is not clear how much of the improvement is due to the proposed model versus the use of superior LLMs. It is unfair to compare their performances with other state-of-the-art baselines that do not use the same LLM in Table 1. The authors should report the performance of their runs using the same backbones as the compared methods for a fair comparison.\n- Table 2/3 have similar issues to Table 1.\n- In the captioning task in Table 1, the METEOR, CIDEr and SPICE metrics are often considered more important. However, the results for these metrics of the proposed method are only slightly superior to or even inferior to the baselines, which raises doubts about the effectiveness of the proposed method.\n- The ablation study in section 4.6 is not sufficiently convincing. There is no explanation of the task being experimented on, and the authors should provide the final results, not just the loss.", "relation_to_broader_scientific_literature": "No.", "essential_references_not_discussed": "None.", "other_strengths_and_weaknesses": "See the above problems.", "other_comments_or_suggestions": "None.", "questions_for_authors": "None.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "RiwItfGGlx", "reviewer": "Reviewer_1cT4", "summary": "The paper proposes a subject-agnostic encoding from fMRI recordings into an LLM space to enable text decoding from brain data. The paper claims this approach generalizes across subjects with different numbers of voxel measurements, and that it outperforms existing baselines.", "questions": "", "limitations": "", "rating": 4, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**C1**\n\nThe text claims UniBrain is subject-agnostic, but Table 1 lists MindLLM as the only subject-agnostic model—this inconsistency needs clarification.\n\n**R1**\n\nWe apologize for the mistake and thank the reviewer for bringing this to our attention.\n\n> with only a few exceptions (Mai & Zhang, 2023).\n> \n\nShould actually be\n\n> with only a few exceptions (Wang et al., 2024b)\n> \n\nThe models in the two references share the same name, which led to the citation error. We will correct it in the revised version.\n\n**C2**\n\nLine 32 \"results\" should be upper-case at the beginning of the sentence. Something is also off about the grammaticality of the sentence.\n\n**R2**\n\nWe thank the reviewer for pointing this out. We have corrected the capitalization and revised the sentence for clarity and grammatical correctness. The updated version reads:\n\n> The results demonstrate that our model outperforms the baselines in downstream tasks, generalization to unseen subjects, and adaptation to novel tasks.\n> \n\n**Q1**\n\nHow well does your model scale with the number of subjects? I.e. how much better does it perform with pre-training on e.g. 3 subjects compared to 1 etc?\n\n**A1**\n\nWe appreciate the reviewer’s valuable question and suggestion. We conducted experiments to evaluate how model performance scales with the number of subjects and report the performances on the COCO caption task. We examined both in-distribution (seen subjects) and out-of-distribution (held-out subjects) settings. \n\n[Table](https://anonymous.4open.science/api/repo/MindLLM-FAA4/file/rebuttal/scaling.png)\n\nOur results show significant performance improvements as the number of training subjects increases, demonstrating that the model benefits from exposure to more subjects during pre-training. We will include more results on other subjects and datasets in the revised manuscript.", "claims_and_evidence": "As written in Table 1, the proposed MindLLM is the only model that is subject-agnostic. But only a few lines above in the text, it is written that UniBrain (Mai & Zhang, 2023) is capable of doing so: \"models for fMRI decoding can not handle varying input shapes and are not subject-agnostic, with only a few exceptions (Mai & Zhang, 2023)\". Which is correct?\n\nModel performance indeed seems superior to baseline methods, with consistent gains across datasets (Tables 1 and 2). The generalization across subjects is what is most convincing to me (Table 3). The model also seems to adapt well to new tasks (Table 4).\n\nOverall, this seems like a good contribution.", "methods_and_evaluation_criteria": "Evaluation methods are fair as I can tell, but I am not an expert in brain-to-text decoding.", "theoretical_claims": "N/A.", "experimental_designs_or_analyses": "Using standard benchmarks as far as I can tell. I cannot judge if the choice of baselines is comprehensive.", "relation_to_broader_scientific_literature": "Decoding text signals from brain data is a broadly studied phenomena, and the use of LLMs for brain data has gained a lot of popularity in recent years. This work adds to that body of literature by proposing a novel subject-agnostic encoder to adapt individual brain recordings into a common LLM space.", "essential_references_not_discussed": "N/A.", "other_strengths_and_weaknesses": "-", "other_comments_or_suggestions": "Line 32 \"results\" should be upper-case at beginning of sentence. Something is also off about the grammaticality of the sentence.", "questions_for_authors": "How well does your model scale with the number of subjects? I.e. how much better does it perform with pre-training on e.g. 3 subjects compared to 1 etc?\n\nedit after rebuttal: authors have run this and results look very promising. Increased score from 3 to 4.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "BDTjD7aGn1", "reviewer": "Reviewer_SE3R", "summary": "This works a versatile and subject-agnostic model for fMRI-to-text decoding. It demonstrated a brain-instruction tuning approach, inspired from visual instruction tuning framework. This model has specifically designed with application across subjects with varied number of recording voxels, which is a common challenging question in the BCI applications. Compared to existing model using an averaging pooling or sampling strategy, this model proposes an neuroscience-informed attention structure that allows information extraction effectively. In the encoder, the query embeddings are learnable, and key embedding integrate positional information. Meanwhile, it  utilized the NSD datasets (MSCOCO datasets) to build the link between visual images and text inputs, which allows diverse and versatile downstream tasks like perception, memory, language, and reasoning.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**C1**\n\nThe manuscript lacks comparisons or references on brain-machine alignment or image/video decoding [1][2][3].\n\n**R1**\n\nWe thank the reviewer for pointing out these relevant references. [1] [2] deal with fMRI time series, while our method focuses on static fMRI (i.e., fMRI signals at a moment). Therefore, they are not comparable to our work. We will create a new section for multimodal brain decoding in related work and discuss these models in the revised version.\nFor [3], we have included it in the baselines. The results are shown below and will be updated in the revised version (The MindVis column is the new baseline).\n\n[Table](https://anonymous.4open.science/api/repo/MindLLM-FAA4/file/rebuttal/table2.png)\n\n**C2**\n\nThe framework is not very novel, as it directly mimics the visual instruction tuning.\n\n**R2**\n\nThe training paradigm of brain instruction tuning is similar to visual instruction tuning. However, the construction of the brain instruction tuning datasets is a nontrivial contribution. As discussed in section 3.3 (lines 177-184, right column),  we select datasets to capture diverse aspects of semantic information embedded in fMRI signals, which are considered among the most fundamental and essential properties of human brains.\n\n**Q1**\nThe computational cost for model's training and inference, compared to other baselines.\n\n**A1**\n\nWe thank the reviewer for this valuable question. We summarize the computational complexity and runtime analysis below.\n\n[Table](https://anonymous.4open.science/api/repo/MindLLM-FAA4/file/rebuttal/cost.png)\n\nExperiments here were conducted on run on an A40. All timings are measured during inference with `batch_size=1`. When comparing with other encoders, we only measure the encoder part (LLM is excluded). We separately time the LLM at the last row.\n\nDespite its superior performance, our model introduces only a marginal increase in encoder-side computational cost compared to other baselines. Notably, the majority of inference time and complexity arises from the LLM component, which is shared across all models. This highlights that the design choices in the encoder—while crucial for performance—do not significantly affect runtime.\n\n**Q2**\n\nDoes scaling laws of data observed in the results, more subjects trained will increase the accuracy?\n\n**A2**\n\nWe appreciate the reviewer’s valuable question and suggestion. We conducted experiments to evaluate how model performance scales with the number of subjects and report the performances on the COCO caption task. We examined both in-distribution (seen subjects) and out-of-distribution (held-out subjects) settings.\n\n[Table](https://anonymous.4open.science/api/repo/MindLLM-FAA4/file/rebuttal/scaling.png)\n\nOur show significant performance improvements as the number of training subjects increases, demonstrating that the model benefits from exposure to more subjects. We will include more results on other subjects and datasets in the revised version.\n\n**Q3**\n\nHow is subject-specific information encoded in this framework?\n\n**A3**\n\nWe do not encourage the encoding of subject-specific information in the method, as our model is designed to be *subject-agnostic*—capable of generalizing to individuals not seen during training without requiring additional fine-tuning. This \"out-of-the-box\" generalization has significant practical benefits: it supports scalable deployment across diverse populations and eliminates the need for costly, subject-specific personalization or calibration.\n\nTo assess how subject-specific and subject-agnostic information evolve through the model, we visualize the latent embeddings at various stages of the encoder using T-SNE below.\n\n[Figure](https://anonymous.4open.science/api/repo/MindLLM-FAA4/file/rebuttal/latent.png)\n\nThis figure illustrates the transition from subject-specific to subject-agnostic representations. Initially, the embeddings exhibit distinct subject-wise clusters, indicating strong subject-specific information. After passing through the neuroscience-informed attention layer, these clusters are still visible. However, as the embeddings propagate through successive MLP layers, the subject-specific patterns gradually dissolve. By the final layer, the latent space forms a well-mixed representation where subject identity is no longer distinguishable.\n\nThis transformation is crucial for enabling generalization to unseen subjects in a zero-shot manner.", "claims_and_evidence": "This framework is subjective-agnostic, as the designed attention modules allow flexible information extraction regardless of the mismatch of recorded voxels, the claim is clearly supported. As it is demonstrated in a comprehensive benchmarks, compared to existing methods, MindLLM achieves the state-of-the-art accuracy in different downstream tasks. Meanwhile, it has also been evaluated on a subject held out settings, and achieving the best accuracy.", "methods_and_evaluation_criteria": "NSD dataset is a well-established benchmark dataset in brain decoding, and MSCOCO dataset is a popular dataset in computer vision, and has been extended into other diverse task settings, including captioning, QA, etc. The evaluations and metrics including BLEU, METEOR are properly utilized. Multiple baselines in this field including MindBridge, UniBrain, BrainChat are compared against.", "theoretical_claims": "This work does not include any theoretical proofs or claims.", "experimental_designs_or_analyses": "This experiment designs are solid, as it has been demonstrated on largest-scale brain decoding dataset recently released, although including other datasets such as BOLD5000 (cross-dataset generalization) could increase more soundness of the evaluation. Multiple established baselines are compared against. An ablation study of critical components in the models has also been implemented in this study.", "relation_to_broader_scientific_literature": "Brain decoding is a challenging field, recent studies start to integrate brain decoding with existing large language models framework to enhance its capability for text decoding. Subject-agnostic decoding is one of the major challenges in this field, and this work is well-motivated to address such issue. This work covers sufficient literature review, is comprehensively grounded with comparisons with multiple existing baseline model, and the results demonstrate certain level of improvement.", "essential_references_not_discussed": "Although this work only focused on text decoding, while it lacks of comparisons or references on brain-machine alignment or image/video decoding from related literatures, for example [1][2][3].\n\n[1] Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain).\n\n[2] Cinematic Mindscapes: High-quality Video Reconstruction from Brain Activity.\n\n[3] Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding.", "other_strengths_and_weaknesses": "This work demonstrate an impactful framework for brain instruction tuning, and allows subject-agnostic decoding, and achieves the state-of-the-art accuracy, while it does not outperforms existing baselines significantly. Meanwhile the framework itself is not novel, as it directly mimic the visual instruction tuning. The framework is lack of capability of integrating the image information, which potentially could be a future extension.", "other_comments_or_suggestions": "N/A", "questions_for_authors": "1. Computational cost for model's training and inference, compared to other baselines.\n2. Does scaling laws of data observed in the results, more subjects trained will increase the accuracy?\n3. How is subject-specific information encoded in this framework?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {"ethical_review_concerns": "N/A"}}, {"review_id": "2OxtM7e9sx", "reviewer": "Reviewer_D9CN", "summary": "A large body of research in visual image processing has focused on either brain encoding or decoding, aiming to understand how the brain processes natural scenes or reconstructs images from human brain activity. Recent fMRI brain decoding studies have specifically targeted advanced brain-computer interfaces, where incorporating natural language instructions and fMRI encoder latent vectors into a LLM decoder provides deeper insights into brain mechanisms. However, previous studies, where models were trained independently for each subject, have resulted in poor generalization across subjects. To address this, the current study introduces a novel approach called MindLLM. MindLLM develops a subject-agnostic fMRI encoder capable of accommodating subjects with varying input shapes, thus achieving high-performance subject-agnostic decoding. Additionally, the authors introduce brain instruction tuning, which enhances the model’s ability to capture diverse semantic representations from fMRI signals and generalize to new tasks. Experimental results on the popular NSD (Natural Scenes Dataset) and comprehensive fMRI-to-text benchmarks demonstrate that MindLLM outperforms baselines in downstream tasks, unseen subject generalization, and novel task adaptation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "**C1** It is unclear how the fMRI encoder handles different numbers of voxels.\n\n**R1** Each voxel is treated as a token in our model, and the attention layer learns to map sequences of varying lengths into a fixed-dimensional representation. This is similar to using multiple [CLS] tokens in a BERT model to capture the semantics of sentences of various lengths. This works because, in attention, the output dimension is solely determined by the number of queries. We also provide a [mathematical clarification](https://anonymous.4open.science/api/repo/MindLLM-FAA4/file/rebuttal/math_clarification.png)\n\n**C2** The idea is not entirely novel—see MindEye2.\n\n**R2** We respectfully disagree with the statement and would like to emphasize the **fundamental differences** between our approach and MindEye2:\n\n1. MindEye2 requires a separate projector layer trained on each subject to map their features into a shared representation space.\n2. The architecture of MindEye2 cannot handle subjects with varying numbers of voxels.\n\nTherefore, MindEye2 cannot generalize to unseen subjects without further tuning on data from the unseen subject. In contrast, our model does not need subject-specific parameters or architectures and can generalize to held-out subjects in a zero-shot manner.\n\n**C3** The brain instruction tuning lacks details of backpropagation.\n\n**R3** The brain instruction tuning trains the model through the loss function in lines 254-255. The gradients are backpropagated from the predicted tokens to the LLM, and all the way back to the fMRI encoder to update its parameter.\n\n**C4** No comparison to show the results w/ and w/o the brain instruction tuning.\n\n**R4** We would like to politely point out that we did have a comprehensive experiment to show the effects of brain instruction tuning (BIT). As shown in Table 2’s caption, models with $\\circ$ are the versions w/o BIT. We can see models w/ BIT significantly outperform their corresponding models w/o BIT. On average, BIT brings 28.0% improvement, as stated in line 340, right column.\n\n**C5** Figure 6 is difficult to interpret.\n\n**R5** We improved [Figure 6](https://anonymous.4open.science/api/repo/MindLLM-FAA4/file/rebuttal/figure6.png)'s clarity by selecting a more perceptually friendly colormap. We also improved the caption to better clarify that each subfigure corresponds to a query token on subject 1. The findings are discussed in section 4.7.\n\n**C6** Brain maps across subjects should be included.\n\n**R6** \nWe thank the reviewer for the suggestion. We now present the query token from Figure 6(a) applied to additional subjects (Subjects 2–7) below.\n\n[Figure](https://anonymous.4open.science/api/repo/MindLLM-FAA4/file/rebuttal/brainmaps.png)\n\nWe observe that the attention maps exhibit highly similar spatial patterns across subjects. This consistency supports the generalizability across subjects to capture common brain patterns.\n\n**C7** The shared vs. subject-specific information in the latent space should be verified.\n\n**R7** We do not encourage the encoding of subject-specific information in the method, as our model is designed to be *subject-agnostic.* This enables scalable deployment across diverse populations and eliminates the need for costly, subject-specific calibration.\n\nTo assess how subject-specific and subject-agnostic information evolve through the model, we visualize the latent embeddings at various stages of the encoder below.\n\n[Figure](https://anonymous.4open.science/api/repo/MindLLM-FAA4/file/rebuttal/latent.png)\n\n**C8** Performances on individual subjects should be included.\n\n**R8**\nWe appreciate the suggestion. The results in Table 2 are based on subject 1. We have now extended our experiments to include subjects 2 and 5 below.\n\n[Table](https://anonymous.4open.science/api/repo/MindLLM-FAA4/file/rebuttal/breakdown.png)\n\nAcross subjects, our model outperforms the baselines in most cases, demonstrating robustness to inter-subject variability. We note that performance varies slightly more on ST-VQA and TallyQA. Models trained on subject 5 outperform those trained on subjects 1 and 2, suggesting that subject 5 provides higher-quality data.\n\nWe will include results for all subjects in the revised version.\n\n**C9** What are the key conclusions from MindLLM? The paper lacks a discussion of the findings and implications of the current work.\n\n**R9** While interpretation (Figure 6) supports our model’s motivation, our primary focus is **improving decoding performance**.\n\nKey findings:\n\n- **Brain instruction tuning** significantly improves the performance.\n- **Neuroscience-informed attention** significantly outperforms vanilla cross-attention (Section 4.6), offering architectural insights for fMRI decoding.\n\nImpact:\n\nOur method enables subject-agnostic decoding and easy task adaptation, which unlocks out-of-the-box applications like prosthetic control without subject-specific finetuning. We will update detailed discussions in the revised version.", "claims_and_evidence": "The submission makes several claims about the effectiveness and innovation of the MindLLM approach, particularly in its subject-agnostic encoding and the ability to generalize across subjects. However, some of these claims are not fully supported by clear and convincing evidence:\n* The authors claim that the encoder is subject-agnostic is made, but the evidence provided is not sufficiently detailed to demonstrate how the shared and unique information across subjects is learned. There is a need for more explicit evidence on how the encoder generalizes across subjects, especially given that different subjects have varying numbers of voxels. The absence of a thorough analysis of this aspect makes the claim less convincing.\n* While the paper presents results on multiple benchmarks, there is no clear distinction between subject-generalizable benchmarks and those that show subject-specific variations. It would be beneficial to see a more detailed analysis of the performance on individual subjects, and whether the model performs similarly across all subjects on certain tasks or whether it varies significantly. This would help substantiate the claim that MindLLM generalizes effectively.\n* The authors claim that brain instruction-tuning enhances the model’s performance, but the details on how this is performed (e.g., whether the loss is backpropagated, and how it affects the LLM decoder’s weights) are not provided. The lack of a comparison between results before and after brain instruction-tuning weakens the claim, as it is not clear how much improvement the tuning provides or what specific changes occur in the model's performance due to this process.\n* The authors claim that the flatbrain maps illustrate important insights from the model is undermined by the difficulty in interpreting the figure due to the colormap choice. Without a clearer visualization and better explanation of what each flatmap represents, particularly in relation to specific subjects and query tokens, the evidence provided in Figure 6 does not strongly support the claims about the model’s effectiveness.", "methods_and_evaluation_criteria": "The proposed methods, including the subject-agnostic encoder and brain instruction-tuning, are suitable for the problem of brain decoding and the application of multimodal language models. However, additional clarifications on how the methods are implemented and evaluated (particularly in terms of how they handle varying voxel counts and the impact of brain instruction-tuning) would enhance the robustness of the approach. The evaluation criteria, including the use of fMRI-to-text benchmarks, are appropriate, but further breakdowns of subject-specific performance would provide a more comprehensive understanding of the model's generalizability.", "theoretical_claims": "Since the paper does not present formal mathematical proofs to substantiate its theoretical claims, the correctness of any such proofs does not apply.", "experimental_designs_or_analyses": "The experimental design of the study is generally sound, but there are several areas that need further clarity and validation. These issues are highlighted in the weaknesses section.", "relation_to_broader_scientific_literature": "The key contributions of the MindLLM approach build upon several important areas of scientific literature, including brain encoding and decoding, brain-computer interfaces, multimodal llms, and cross-subject generalization.", "essential_references_not_discussed": "Yes, all the relevant works are clearly discussed in the current version, and the related work section is adequate.", "other_strengths_and_weaknesses": "Strengths:\n1. The concept of learning a query weight matrix to transform each subject's fMRI data into tokens is intriguing, as this approach can be generalized for new subjects with varying input shapes.\n\n2. The method of handling position encoding for key vectors, particularly the use of Fourier transformers for voxel coordinates, is well-handled.\n\n3. The proposed MindLLM is interesting, as it leverages fMRI-to-text benchmarks and utilizes fMRI encoding vectors passed to an LLM decoder, making MindLLM an fMRI instruction-tuned multimodal large language model.\n\nWeaknesses:\n\n1. There are several significant weaknesses in this work, particularly regarding the methodology and experimental results:\n   * The majority of the results presented in this study focus on fMRI-text decoding benchmarks. However, the underlying methodology—the subject-agnostic fMRI encoder—has not been sufficiently explained. Specifically, how was this approach implemented, and was the latent space verified for each subject? Given that fMRI recordings have low signal-to-noise ratio (SNR), it is crucial to understand how the common latent space functions. Furthermore, whether the transition from the latent space to the subject space is accurately reconstructing the voxels and regions is a key issue. Without proper validation of this, it is difficult to trust the results of the fMRI-text decoding.\n   * Additionally, the authors should clarify their subject-agnostic fMRI encoder approach in more detail. With the current explanation, it is unclear how the same query weight matrix is learned across subjects. For example, if one subject has 12,000+ voxels and another has 15,000 voxels, how is the same weight matrix used to project data into a common space? If I misunderstood, the authors should provide a clearer explanation of this methodology.\n\t\n2. The concept of Mind-LLM with a subject-agnostic encoder is not entirely novel. It seems quite similar to the approach presented in the MindEye-2 paper, where the authors also learn a subject-agnostic encoder and use it to generalize on held-out subjects. Therefore, the authors of the current study could provide a clearer explanation of how the MindLLM approach differs from MindEye-2, especially considering that both papers use the same NSD dataset. \n\n3. There are no details provided on how the authors perform brain instruction-tuning. Specifically, it is unclear whether the loss is backpropagated during brain instruction-tuning. If so, how are the weight parameters in the LLM decoder affected or updated? Additionally, there is no comparison of the results before and after brain instruction-tuning to demonstrate its impact.\n\n4. Since the authors have learned a subject-agnostic encoder, it would be helpful to understand what specific shared information is learned across subjects and what unique information is learned for each subject. Additionally, as the authors perform various benchmarks, it would be insightful to know if any benchmarks exhibit similar performance across subjects, while others show subject-specific variations in results.\n\n5. What are the key conclusions from MindLLM? The flatbrain maps in Figure 6 are difficult to interpret due to the choice of colormap. Additionally, there are six flatmaps, and it is unclear whether each subfigure corresponds to a specific subject. The authors mention that each subfigure is related to a query token, but this makes Figure 6 difficult to read and interpret. As a result, drawing meaningful conclusions from this figure is challenging.\n\n6. The paper lacks a discussion of the findings and implications of the current work.", "other_comments_or_suggestions": "* Clarity of Figures: Some figures, especially Figure 6 (flatbrain maps), could benefit from clearer labeling, improved colormap choices, and better explanations to enhance readability and interpretation.\n* The authors could consider including subject-specific and shared variance brain maps to further illustrate the model's ability to generalize across subjects and capture common brain activity patterns.", "questions_for_authors": "* The questions raised in the weaknesses section regarding the subject-agnostic encoder, brain instruction-tuning, and performance across subjects would benefit from further clarification. \nPlease refer to the details discussed in that section for more context.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This is a paper presenting a cross-subject question answering model that analysis fMRI patterns and answers questions about what was being processed or intended at a given time. The authors perform modeling at a voxel / parcel level use a neuroscience informed attention process where they replace the attention key with the location in the brain (specific regions have more or less specific functions, at least we know that neighboring voxels have similar functions). The authors also introduce a brain instruction. The authors show that their model has good performance relevant to baseline, and the more interesting part is that it is agnostic to subject identity and that it performs better using combinations of data across subjects (the authors are requested to add these results to the paper). The authors are also requested to add all their answers to the comments in the paper as well as the other results and changes they promised.
Decision: Accept (poster)