Correlating instruction-tuning (in multimodal models) with vision-language processing (in the brain)
Authors: SUBBA REDDY OOTA
Akshett Rai Jindal
Ishani Mondal
Khushbu Pahwa
Satya Sai Srinath Namburi GNVV
Manish Shrivastava
Maneesh Kumar Singh
Bapi Raju Surampudi
Manish Gupta
Venue: ICLR 2025 Poster (2025)
Keywords: brain encoding
fMRI
visual processing
multimodal instruction-tuned models
language decoder
LLMs
MLLMs
Primary area: applications to neuroscience & cognitive science

Abstract: Transformer-based language models, though not explicitly trained to mimic brain recordings, have demonstrated surprising alignment with brain activity. Progress in these models—through increased size, instruction-tuning, and multimodality—has led to better representational alignment with neural data. Recently, a new class of instruction-tuned multimodal LLMs (MLLMs) have emerged, showing remarkable zero-shot capabilities in open-ended multimodal vision tasks. However, it is unknown whether MLLMs, when prompted with natural instructions, lead to better brain alignment and effectively capture instruction-specific representations. To address this, we first investigate the brain alignment, i.e., measuring the degree of predictivity of neural visual activity using text output response embeddings from MLLMs as participants engage in watching natural scenes. Experiments with 10 different instructions (like image captioning, visual question answering, etc.) show that  MLLMs exhibit significantly better brain alignment than vision-only models and perform comparably to non-instruction-tuned multimodal models like CLIP. We also find that while these MLLMs are effective at generating high-quality responses suitable to the task-specific instructions, not all instructions are relevant for brain alignment. Further, by varying instructions, we make the MLLMs encode instruction-specific visual concepts related to the input image. This analysis shows that MLLMs effectively capture count-related and recognition-related concepts, demonstrating strong alignment with brain activity. Notably, the majority of the explained variance of the brain encoding models is shared between MLLM embeddings of image captioning and other instructions. These results indicate that enhancing MLLMs' ability to capture more task-specific information could allow for better differentiation between various types of instructions, and hence improve their precision in predicting brain responses.

Published as a conference paper at ICLR 2025
CORRELATING INSTRUCTION- (IN MULTI- MODAL MODELS) WITH VISION-LANGUAGE PROCESS- ING (IN THE BRAIN)
Subba Reddy Ootal*, Akshett Jindall Ishani Mondal Khushbu Pahwa
Satya Sai Srinath Namburi', Manish Shrivastava', Maneesh Singh
Bapi S. Raju", Manish Gupta?
Techhniche Universität Berlin, Germany, 2IIT Hyderabad, India, Univ of Maryland, USA
"Rice Univ, USA, Univ of Wisconsin- Madison, USA, SSpector Inc, USA
Microsoof, India
subba reddy. www.ato-beeerinnneee gmanishtmiacroooft.com
ABSTRACT
Transformer-based language models, though not explicitly trained to mimic brain recordings, have demonstrated surprising alignment with brain activity. Progress in these models-throupe increased size, instruction-tuning, and multimodal- ity-has led better representational alignment with neural data. Recently, a new class of instruction-tuned multimodal LLMs (MLLMs) have emerged, showing remarkable zero-shot capabilities in open-ended multimodal vision tasks. How- ever, it is unknown whether MLLMs, when prompted with natural instructions, lead to better brain alignment and effectively capture instruction-specifi repre- sentations. To address this, we first investigate brain alignment, i.e., measuring the degree of predictivity of neural visual activity using text output response em- beddings from MLLMs participants engage in watching natural scenes. Ex- periments with different instructions (like image captioning, visual question answering, etc.) show that MILMs exhibit significantly better brain alignment than vision-only models and perform comparably to non-in non-instruction tuned mul- timodal models like CLIP. We also find that while these MLLMS effective generating high-quality responses suitable to the task-specific instructions, not all instructions are relevant for brain alignment. Further, by varying instructions, we make the MLLMs encode instriction-specifie visual concepts related to the input image. This analysis shows that MLLMs effectively capture count-related and recognition-relater concepts, demonstrating strong alignment with brain activity. Notably, the majority of the explained variance the brain encoding models is shared between MILM embeddings of image captioning and other instructions. These results suggest that enhancing MLLMs' ability to capture task-specific in- formation could lead to better differentiation between various types of instruc- tions, and thereby improving their precision in predicting brain responses. We make the code publicly available
INTRODUCTION
Brain encoding aims at constructing neural brain activity recordings given an input stimulus. Prior studies brain encoding have demonstrated that representations from multimodal models, which align multiple modalities (e.g., vision and language), achieve higher degree of brain alignment for both image- based and text-based representations compared to vision-oni only models, particularly when using naturalistic image stimuli (Doerig et al.] 2022 Wang al. 2022 Oota al.) 2022b Popham et al. 2021] Oota et al. 2022bb.. Specifically, these studies have shown that representations from multimodal models are better at predicting neural responses in the high-level visual cortex as compared to previous vision-only models like convolutional neural networks (CNNs) (Wang et al.
http:///ggggtt...ommm//aaaaer/// 1m_instruct: on_brain

---

Published as a conference paper at ICLR 2025
Captioning Ws soption ofe
Task-speciffi
Instruction Embedding
Vinaal Rag ora
Nuttimodal
Estirnate
allgnment (CHLLA))
image Visusl Question Ansvering Are people this mage? yes, describee them.
(MIL)M)
NSD naturalistie Image atimulus
Task-spacific instructions
Figure Leveraging instruction-tuned multimodal LLMs for brain encoding with diverse set of instructions. For the given image, we could obtain different multimodal representations using instructions that ask the model to (i) generate the caption of the image, (ii) identify whether people are present, or (iii) determine the primary colors dominant in the image. Using instruction-specine representations, we estimate the alignment using a simple linear function f (ridge regression) which map MLLM representations to brain recordings.
2022 Popham et al., 2021). However, prior research investigating the effectiveness of multimodal models for brain alignment has primarily relied on image-caption pairs, leaving the full potential of multimodal models, especially when enhanced with large language models (LLM) and task-specific instructions, underexplored. In this paper, we explore how effectively multimodal representations, obtained by prompting MILMs with various natural instructions, align with visual processing brain
regions.
Several previous unimodal studies have found that Transformer models finetuned for specific tasks more closely align with brain processes during language comprehension (Oota et al. 2022a Aw
& Toneva 2023 Sun & Moens 2023; Oota et al. 2024c), speech processing (Otta et al. 2023
Tuckute et al. 2023; Oota et al. 2024a), and visual processing (Wang et al. 2019 Conwell et al, 2022). yielding better brain predictivity than pretrained models. However, these studies rely on separate task-specific models, which limits generalization, as each task-specific representations are aligned with the same human brain recordings. Furthermore, finetuning data needs be obtained for each task, and a new model needs to be trained separately. Recently, instruction-tume tuninng has become a widely adopted method for fine-tuning the same baseline large language model across multiple different natural language processing (NLP) tasks. This approach has been shown to outperform task specifie models (Taori al., [023 Touvron et al. [2023] Diang et al., 2023 [Abdin et al. [202]] Dubey et al. [0024) and demonstrates improved brain alignment compared to smaller language
models (Sun et al. 2023, Sun Moens 2023 Loong Aw et al. 2024).
Building on these advances, researchers have extended instruction-tuning to multimodal LLMs (MLLMs) (Xu et al. [202]] Dai et al., 2023] Liu et al., 2024), enabling impressive unimodal and multimodal capabilities. These progressive improvements motivate to explore the effectiveness of instruction-tumm tuned MLLMs for brain encoding. By leveraging task-specific representations from single MILM, we aim to capture multiple aspects of image beyond simple captioning, such as people, foreground and background elements, interactions between objects, environments, col- ors, food items, animals, and outdoor scenes. This leads to critical question related under- standing of human-alignment of AI: Do these multimodal instruction-tuneet models prompted using natural language instructions lead to better brain aligument and differentiate instruction-sseciis representations? To address this, we investigate different ways of prompting MLLMs with various task spec-iie instructions. Overall, this research utilizes various task-specific MLLM representations to develop encoding models based on fMRI responses within a multimodal model framework (see Fig. [ for workflow).
Using brain recordings of participants watching natural scenes images from the NSD dataset (Allen et al., al. 2022), we investigate several research questions. First, we explore the effectiveness of MI..M representations and compare their brain alignment with unimodal and multimodal models. For the purposes of this work, we focus on three MLLMS (InstructBLIP (Dai et al.) 2023), mPLUG- Owl (Ye et al. 2023) and IDEFICS (Laurençon et al. 2023)), one bmage-based (ViT-H) and one multimodal model (CLIP). We probe these MLLMS using ten different instructions across six vi- sual tasks. Specifically, we investigate which of these task-specific instructions result in better brain alignment. Second, do the instruetion-speccfic representations from MILMS differentiate the visual regions that process this information, thereby aligning with the mechanisms of human visual cogni- tion? Third, do task-specific instructions from MLLMs account for visual concepts understanding,
2

---

Published as a conference paper at ICLR 2025
and which brain regions responsible for processing different visual concepts? Fourth, we use variance partitioning approach to assess the unique and shared variance of each task-specific instruc- tion to brain responses. This analysis provides insights into how different visual tasks complement or overlap in explaining brain activity, thereby enhancing our understanding of the functional orga- nization of visual processing in the brain.
Our analysis of instruction-tunee MLLMS and brain alignment reveals several key conclusions: (i) MLLMs demonstrate significantly better brain alignment than vision-only models and perform com- parably to multimodal models such as CLIP. (ii) While all three MLLMS generate task-specific out- put tokens based on instructions, not all instructions contribute to higher brain alignment. Specif- ically, the image captioning instruction leads to stronger brain alignment in regions like the EBA (extrastriate body area), PPA (parahippocampal place area), and FFA (fusiform face area), whereas instructions related to image understanding result in higher alignment in early visual regions. (iii) Furthermore, while MLLMs capture several visual concepts, such as counts and recognition, for other concepts like color, positional understanding, and general scene understanding, MLLMs ex- hibit similar brain alignment patterns. (iv) By employing a variance partitioning approach, we find that most of the variance is shared across instructions, with high overlap between Image Caption- ing (IC) and other prompts but lower overlap with image understanding and scene recognition. This suggests that MILMS could improve in differentiating between various types of instructions.
Overall, our work is the first to propose the use of instruction-tunel MILMs and to demonstrate the differences in task-specific representations within MILMs, along with the reasons behind these differences in relation to brain alignment. We make the code publicly available
2 RELATED WORK
Brain encoding using multimodal models. The human brain perceives the environment using in- formation from multiple modalities (Gauthier et al.] 2003). Therefore, examining the alignment between language and visual representations in the brain by training encoding models on fMRI responses, while extracting joint representations from multimodal models, can offer insights into how our brain processes multimodal information. For instance, it has been shown in Doerig et al. (202);; Wang al. ): Oota al. (022b);; PPopham et al. (202T) that multimodal models like CLIP [Radford et al. 2021b) better predict neural responses in the high-level visual cortex as com- pared to previous vision-only models. Additionally, Tang et al. (2024) demonstrate the use of mul- timodal models in a cross-modal experiment to assess how well the language encoding models can predict movie-fMRI responses and how well the vision encoding models can predict narrative story- fMRI. Nakagi et al.) (2024) analyzed fMRI related to video content viewing and found distinct brain regions associated with different semantic levels, highlighting the significance of modeling vari- ous levels of semantic content simultaneously. However, these studies have primarily focused on multimodal models aligned embedding space when text captions were provided as input, leav- ing the new class of instruction- tions-stie still unexplored. Unlike previous work, we are the first to study multimodal instruction- models with language-guidee guided instructions and to perform comprehensive brain alignment analysis while subjects are engaged in viewing passive images.
MLLMs-whin which utilize task specific natural language instruc-
Task- brain alignment. Our work is also closely related to that of Wang et al. (2019); Oota et al. (2022ab; Aw & Toneva (2023); Sun et al. (2023) and Aw et al. (002)) who propose using task- specific model representations to study the contribution of individual tasks brain alignment, Wang eet al. (2019) investigated 21 computer vision tasks to explore which vision tasks are more a aligned with the brain while subjects engaged in viewing passive images. Similarly, Oota et al. (2022a) and Sun et al. (2023) explored 10 GLUE NLP tasks to study which NLP tasks are more brain- train-aligned during reading and listening to stories. Aw & Toneva (2023) further extended the comparison by evaluating pretrained models that were trained either on web data or BookSum stories to determine whether BookSum models provide better character-specific information in brain alignment tasks. More recent work by Aw et al. (2023) uses instruction-tuned language models to investigate the effect of natural language instruction model representations on brain alignment across layers for language comprehension. We complement these works by examining the impact of a wide range of multimodal instruction- tuned models on brain alignment and by studying the effect of tpek-fi,, language- guided instructions from MLLMMs on their alignment with brain activity.
3

---

Published as a conference paper at ICLR 2025
3 DATASET AND CURATION
Brain dataset. We use the Natural Scenes Dataset (NSD) introduced by Allen et al. (2022). which contains high-quality IMRI readings of 8 participants watching images from the COCO dataset (Lin al,, 2014). We analyzed brain data for participants (who completed all the sessions) where each participant was presented with 10, 000 images in total (1, 0.0 common for all participants and 9, 000 unique for a participant). Each of the 10, 000 images were repeated three times in ran- dom order, giving total of 30, 000 readings per participant. Símilar to prior studies (Scotti et al.) 2024a), we compute the mean of the fMRI readings for all the three occurrences and obtain a one- to-one mapping between the image and the corresponding fMRI reading. The images belong to 12 categories: animals, accessories, appliances, electronics, food, furniture, indoor, kitchen, outdoor, person, sports, and vehicles.
The dataset is already pre- pre-processed and a brain mask for each subject is also provided to obtain the activation voxels. Similar to prior studies (Scotti et al.. 2024a b). we use preprocessed flattened IMRI voxels in 1.8-mm native volume space corresponding to the "nsdgeneral" brain region, defined by the NSD authors as the subset of voxels in posterior cortex most responsive to the visual stimuli presented (between 13,000 to 16,000 voxels per participant). We perform the ROI (region of interest) analysis for the NSD dataset considering the following five visual processing regions: body-selective regions (floc-bodies), face-selective regions (floc-faces) and scene-selective regions (floc-places, word-selective regions (floc-words), and pRF- Visual ROIs (also called Retinotopic Early Visual Cortex) (Allen al. 2022). Note that floc-bodies, floc-faces, floc-places and floc-words are high- level visual areas while Vis-alsual ROIs are early visual areas. We list the detailed sub-ROIs of these ROIS in Appendix B
Estimating dataset cross-subject prediction accuracy. To account for the intrinsic noise in bio- logical measurements, adapt (Schrimpf et al. (2021)ss method to estimate the cross-subject pre- diction accuracy for a model's performance. By subsampling fMRI datasets from 4 participants, generate all possible combinations of participants (s [2,4)) watching natural scenes, and use a voxel-wise encoding model (see Sec. 5) to predict one participant's response from others. Note that the estimated cross-subject prediction accuracy is based on the assumption of a perfect model, which might differ from real-world scenarios, yet offers valuable insights into model's performance. We present the average cuose-subject prediction accuracy across voxels for the NSD dataset in Ap- pendix [ [8] The figure suggests that the cross-subject prediction accuracy is consistent across subjects, indicating that all subjects share a similar amount of explainable variance. Cross-subject prediction accuracy for each participant brainmaps are reported in Appendix C in Fig. g
METHODOLOGY
Instruction-tuned Multimodal large language models. To investigate whether multimodal instruction-tunel models prompted using natural language-guided instructions perfectly align with the way humans process visual information in the brain, we consider three popular modern instruction-tuned multimodal models publicly available Huggingface (Wolf et al., 2020): In-
structBLIP DDai et al., (023), mPLUG-OWl (Ye et al. 2023) and IDEFICS (Lurençonn et al. 2023)
InstructBLIP (Dai et al. 2023) is a vision-language instruction-tuned model built upon the pre- trained BLIP-2 model (Li et al. 2023). mPLUG-OWl (Ye et al., (023) is an MLLM designed to perceive and integrate multiple modalities (visual and language) while considering visual context and information to generate corresponding outputs. IDEFICS (Laurençon et al. 2023) is an MLLM based on Flamingo (Zhu et al.) 2024), which accepts arbitrary sequences of image and text inputs and generates text tokens. All MILMS consist of 32 layers and produce 4096-dimensional repre- sentations at each layer. We provide more details, including model-parameters and training dataset details in Table 2 in Appendix
Natural instructions. To ensure the diversity of task-specific instructions while considering image as input, we consider 10 instructions, and extract the language-guided representations from multi- modal instruction-tumm models. As shown in Table [ the 10 natural instructions cover 6 task cat- egories, including image captioning, visual question answering, visual relationships, commonsense reasoning, image understanding and scene recognition. These set of 10 instructions are inspired from the list of 62 multimodal tasks defined in Multilnstruct (Xu et al. 2023). We borrowed those
4

---

instruction. Published as a conference paper at ICLR 2025
Table 1: Instructions for various multimodal tasks (ordered by complexity, from least most com- plex.)
Task
Description
IUT: Deseribe most dominant color image
Image Understanding
IU2: List any food items visible.
IU3: How many animals are there in the image?
VQI: What is in this image?
Visual Question Answering
VQ2: Are there any people in this image? If yes, describe them.
VQ3: What
foreground of the image? What the background?
Image Captioning
IC: Generate some text to descrihe the image
Sccne Recognition
SK: Highlight the arca that shows a natural outdoor scene.
Commonsense Reasoning
type of environment shown image?
Visaral Relationship
VR: What kind of interaction huppening between the animale and inanimate objects bere?
tasks which are generally applicable to any image regardless of the contents in the image. We pro- vide sample of generated outputs for the three MLLMs across 10 instructions in Tables and in Appendix
Extraction of features from instruction-tuned multimodal models. To extract instruction- specific representations from multimodal instruction tunee models for the brain encoding task, we input image and task instruction to obtain the embeddings for the language- We perform zero-shot inference on these models. We use the generate option because the hid- den states are influenced by both the input tokens and the generated tokens, making them dependent on the generation context. The hidden states are dynamic and continuously evolve until the model generates the final token, which involves multiple forward passes in the process. For all multimodal instruction-tur -unnid models, we use the pretrained Transformer weights, which generate hidden state representations at each layer. We then average these hidden state representations of the output gen- erated tokens to obtain the final embedding for each image with respect to the task instruction. For uniformity of feature dimensions, we applied PCA to reduce the dimensions to 1024.
Unimodal and multimodal models. As baseline for comparison, we also included unimodal model ViT-H (Dosovitskiy et al. 2020) and the multimodal model CLIP (Radford et al. 202 which is an image-text alignment model, in our experiments. The pretrained ViT-H model outputs image representations from different layers, providing a 1024-dimensional feature vector across 32 encoder layers. For the CLIP model, input both image and ground truth caption pairs and extracted 1024-dimensional representations from the CLIP-Text model.
5 EXPERIMENTAL SETUP
Voxelwise encoding model. We estimate the brain alignment of multimodal and unimodal models of a image stimulus via training standard voxel-wise encoding models (Deniz et al., 2019; Toneva & Wehbe 2019 Schrimpf et al. 202T). We train bootstrap ridge regression based voxel- -xxe--wiee& encoding models (Deniz et al.) 2019) to predict the fMRI brain activity associated with the stimulus represen- tations obtained from the multimodal instruction-tunee models as well as other models, Formally, for each subject, we encode the stimuli as X; E RD and brain region voxels Y; E IRV, where D denotes the dimension of the image representations, and V denotes the number of voxels. Overall, with N such training images, we obtain N training examples.
Train-Test dataset split. We built an encoding model for each subject as follows: We used all data samples from 9,000 natural images (unique to each subject) for training and tested generalization on samples from the test set of 1,000 images (common across all subjects). Overall, we created per- subject, voxel encoding models. Model details and hyper- parameter settings are in Appendix
Evaluation metrics. We evaluate our models using Pearson Correlation (PC) which is a standard metric for evaluating brain alignment (Jain & Huth 2018 Schrimpf et al., 2021: Gollsteii et al.) 2022 Let Nte be the number of images in the test set. Let Y = (Yi); and Y ( denote the actual and predicted value vectors for a single voxel. Thus, Y and Y E N.. We use Pearson Correlation (PC) which is computed as corr(Y, Y) where corr is the correlation function. The final measure of model's performance is obtained by calculating Pearson correlation between the model's predictions and neural recordings. This correlation is then divided by the estimated cross-subject prediction accuracy and averaged across voxels, regions, and participants, resulting in
Ne
5

---

Published as a conference paper at ICLR 2025
IndtvciBIPP
Intiilizzzion
Irsk MPLUI-OWI
Normalood brain allghiment
alggureen
0.9
Nornalted brain alignmeet
0.8
0.7
0.7
0.6
0.6
0.6
0.5
0.3
0.
6.4
0.
0-3
0.3
0.2
0.2
0.
0.1
(8) Whole Viuall Cortex
(b) cRF-iiual ROTs
[C) PLOC-
Figure 2: Whole visual cortex and ROI-based normalized brain alignment was computed by aver- aging across participants, layers, and voxels. Blue: Average across random initialization the MLLMs. Note that CLIPPttet model uses golden oracle captions while instruct models use predicted model generations. * indicates cases where MLLM embeddings are statistically significantly better than randomly initialized models, i.e., pz 0.05. A indicates cases where MILMs significantly better than unimodal vision models (ViT-H), i.e., ps 0.05. Other brain ROI plots are reported in Fig. TO in Appendix F
standardized measure of performance referred to normalized brain alignment. For calculating normalized alignment, we select the voxels whose cross-subject prediction accuracy 2 0.05.
Variance partitioning. To disentangle task-specific instruction representations from multimodal instruction-tuned models, we used a variance partitioning approach (de Heer et al. [017 LeBel al. 2021). This method measures the overlap in brain variance explained by different task-specific instruction representations. Specifically, variance partitioning separates the brain response variance that can be attributed two models based on their unique and overlapping contributions (Vaidya et al. 2022 Deniz et al., 2019). To perform this, for every pair of instruction representations, we fit separate encoding models for each space well as joint encoding model, obtained by concatenating the features. Using set arithmetic, we can then derive the size of the intersection (NB (N ( where NBA refers to normalized brain alignment, denotes alignment of model 1, (N A)² denotes alignment of model 2 and (NBA) denotes aligmment of the joint model. Similarly, the unique contribution of model I's feature space is computed as (NN
refers to a specific voxel, (N
6 RESULTS
6.1 MLLM REPRESENTATIONS ALIGN WELL TO HUMAN BRAIN ACTIVITY
First, we examine the brain alignment by measuring the similarity of degree of brain predictivity using representations extracted from multimodal instruction-tuned models (MLLMs), focusing both whole visual cortex and specific visual function localizers. For each MLLM, we calculate the average normalized brain alignment across 10 instructions, multiple subjects, and different MLLM layers, using the NSD dataset. Additionally, we report baseline performance using randomly ini- tialized versions of the InstructBIIIP, mPLUG-OWL. and IDEFICS models. We further compare the brain alignment performance of these MLLMs with unimodal vision model (ViT-H) and multimodal CLIP-Text model.
Whole visual cortex analysis. Fig. (a) presents the average normalized brain alignment for whole-brain analysis across six different settings. The results demonstrate that MLLMs signifi- cantly outperform randomly initialized models in terms of brain alignment. Moreover, we find that multimodal intructiio-tuning improves brain alignment over unimodal VYT-H models. Notably, the superior performance of pretrained MI.LMs compared to randomly initialized models indicates that natural guided instructions yield more brain-relevant representations, leading to greater alignment with brain activity. Additionally, MLLMs perform par with, or better than, the CLIP-Text model, despite the latter using ground-truth captions, whereas MLLMs (InstructBLIP, mPLUG-OWL and IDEFICS) mean pooling over predicted output tokens based natural instructions.
ROI analysis. We further examine how citructioo-tuning enhances the alignment of MLLM rep- resentations with brain activity, focusing specific brain regions. To do this, compute the normalized brain aligmment separately across five visual functional localizers, as discussed Sec- (b) and (c) show the normalized alignment across six settings for Visu- ROIs and tion 3 Fig. 2
Visual

---

Published as a conference paper at ICLR 2025
cic righliht the arca cutcoer scors.
ruzs Howe entinats thvan??
Drsss beu does nunt
cn What opea of ronmars tmagg??
ann inter action belwoen animatt innnimmttd
Fongground Background
thea paople ivi imago?
VCu: sinus in the imaget
InstructBLIP
mPLUG-Owl
irage
Figure 3: Each voxel is color coded with the instruction (out of 10) that led the highest normalized brain alignment. The color bar highlights color codes for each instruction. The voxels are projected onto the flattened cortical surface of a representative subject (subject S1) for two MLLMs. Similar brain maps for other subjects in Appendix G
LH
RH
LH
RH
NU:
vQ2:
U2:
vQL:
VQ2
Visual concept: Counts
Visual concept: Recognition
Figure 4: Voxels specific visual concepts groups: Counts (Left) and Recognition (Right). The color bar from Fig. 3 highlights color codes for each instruction. The voxels are projected onto the flattened cortical surface of a representative subject (subject S1).
FLOC-PLACES ROIs (see additional figures in Appendix F.. From Fig. 2 (b) and (c), we make the following observations: (1) Similar to the whole visual cortex analysis, MLL.Ss exhibit significantly better brain alignment compared baseline (randomly initialized) and unimodal VIT-H models. (2) Interestingly, baseline performance is closer to that of unimodal models in early visual ROIs (i.e. Pis-Visual ROIs), whereas the difference becomes more pronounced in floc-place regions. This trend is also observed in MLLMs, where the normalized alignment reaches ~0.8 in place regions but drops ~0.6 in early visual ROIs, suggesting that high-level visual areas provide better brain- relevant representations in MILMss than early visual areas. This finding agrees with previous work which suggests that multimodal models better predicts neural responses the high-level visual cor- tex than previous vision-only models like CNNs (Wang et al. (022). Similar patterns are observed in other high-level visual areas (faces, bodies, words), as detailed in Appendix
6.2 ENCODING PERFORMANCES OF
INSTRUCTIONS FROM MLLMS
While MILMM representations at both whole visual cortex level and in visual functional ROIs demon- strate that multimodal instruction-tunee models improve brain alignment for task-specific instruc- tions, we are also interested in understanding the importance of each task instruction and conducting a layer-wise analysis examine the trends brain alignment across different models.
Which task-specific instructions are highly correlated to visual function localizers? To inves- tigate which instructions are more effective in predicting particular visual functional localizers, we analyze the voxels as follows. For each voxel, we select the instruction that results in the highest normalized brain aligmment and apply the instruction- specific color code to the voxel. The color- scheme corresponding to each instruction is reported in Fig. 3 The figure also displays brain maps for the InstructBL.IP and mPLUG-OWI. for Subject 1, where the voxel color codes are projected onto the flattened cortical surface of the representative subject. Similar brain maps for other subjects are in Appendix G
From Fig. 3 we make the following observations: (i) Representations related to image captioning show higher brain alignment in the EBA, PPA, and FFA regions for both InstructBLIP and IDEFICS models, while the mPLUG-OWL model demonstrates higher alignment mainly in the EBA region. (ii) Instructions related to image understanding (e.g., "describe the most dominant color") result in higher brain alignment in early visual regions for both the InstructBIIPP and mPLUG-OWL. models,

---

Published as a conference paper at ICLR 2025
Layer depth
RH
S1: InstructBLIP
ST: mPLUG-Owl
S1: IDEFICS
Figure 5: Each voxel is color coded with the MILLM layer number (out of 33) that led to the highest normalized brain alignment. The color bar highlights color codes for each layer. The voxels are projected onto the flattened cortical surface of a representative subject (subject SI) for three MLLMs. Similar brain maps for other subjects are in Appendix H
while IDEFICS shows only marginal alignment with this instruction. (iii) Visual question-answering instructions, such as "are there people in this image?" and "foreground and background of the im- age," contribute significantly to brain alignment in higher visual regions, particularly in the PPA and FFA regions. (iv) In contrast to the above three instructions, other instructions (such as "list any food items available" and "how many animals are there?") result into some voxels with high alignment predictions, while the remaining instructions do not result in voxels with significant alignment.
In order to draw some more insights of why some instructions might be more pronounced than other in encoding the brain activity, we clustered instructions into five visual concepts 1) Count 2) Recog- nition 3) Color/Textur 4) Positional Understanding and 5) General Scene Understanding. This is further motivated by our interest in uncovering which brain regions are responsible for different visual concepts.
Do task-specific instructions from MLLMs account for visual concepts understanding? To examine whether task-specific instructions from MLLMs capture different visual concepts and to understand how these concepts are processed in the brain's 's visual regions, we group the instructions into five categories: (i) Count (counting objects, animals. people within the image) - IU2, IU3, VQ2; (ii) Recognition (recognizing objects, animals, or people) VQ1, VQ2: (iii) Color/Texture IU1; (iv) Positional Understanding (foreground/backgroundd right/left, upper/bottom positions)- vQ3; and (v) General Scene Understanding CR.
Similar to task seec--pecific instructions, we analyze the voxels from visual concept groups as follows. We use the same color scheme corresponding to each instruction reported in Fig. The Fig. 4 displays brain maps for the InstructBLIP model for Subject I, with voxel color codes projected onto the flattened cortical surface of the representative subject for two concept groups: Count and Recog- nition. We make the following observations: (i) For the visual concept Count, the VQ2 instruction results in higher brain alignment in high-level visual regions, while IU2 and IU3 instructions show higher alignment in early visual regions. This suggests that MILMS capture count-related visual concepts effectively across instructions, showing alignment with brain activity. (ii) For Recognition, both VQI and VQ2 instructions exhibit distributed brain alignment across both high-level visual regions and early visual regions. (iii) In contrast to Count and Recognition, for visual concepts like Color, Positional Understanding, and Scene Understanding, MLLMs display similar brain align- ment patterns, irrespective of the specific visual concept. Therefore, further improvements may be needed for MILMs to achieve better specificity in processing a broader range of visual concepts. Similar brain maps for other subjects are in Appendix G
Which layers of MLLMs are highly correlated to visual function localizers? To explore whether the effectiveness of layer-wise representations from MLLMs varies in relation to visual functional localizers, we analyze the voxels as follows. For each voxel, we select the layer that results in the highest normalized brain alignment and apply a color code for the 33 layers across the three MLLMs. Fig. pipresenss brain maps for the InstructBLIP, mPLUG-Ow/ and IDEFICS, where the voxels with their corresponding color codes are projected onto the flattened cortical surface of the representative subject (Subject 1). Similar brain maps for other subjects in Appendix H From Fig. 5 we make the following observations: (i) Across all functional localizers, the middle layers of the InstructBLII and IDEFICS models show greater brain alignment for higher visual regions, whereas the later layers
8

---

Published as a conference paper at ICLR 2025
VQ1
V02
V23
VR
CR
1U1
108
SA
10
0.446
0.298
0.87
0.869
0.437
0.392
0.84
0.088
0.983
0:448
0,410
0,363
0.362
0.338
0:335
6.588
0.343
0.306
0.917
0.528
V03
0.371
0.963
0.288
0.315
0.347
0.394
0.288
0.314
0.309
VR
0.863
0.352
0.39:
0.810
0,302
0,625
2.29
0.313
0.30
GR
0.417
0.300
0:347
0.360
0:329
1.363
0.538
IU2
0.311
0.534
0.305
0.200
0.250
0.200
IU3
0.96
0,569
0.357
0.514
0.319
0.365
0.590
0.206
0.290
SR
0:969
0,363
0.228
0.808
00300
0.588
0.330
0.289
0.298
Figure 6: Shared explained variance between pairs of task-specific instructions. higher shared variance indicates that both instructions share similar features, resulting in greater shared explained variance the brain. The diagonal cells are empty since the shared variance between pairs of the same task is identical.
more aligned with early visual regions. In contrast, the later layers of the mPLUG-Owl model result in higher brain alignment for both higher and early visual regions. This variation across the 3 MLLMs may be due to the difference in the underlying language decoder models, which generate output tokens, capture contextual representations, and influence the alignment trend across the layers. This finding is consistent with studies on brain alignment in language models, which have shown that middle layers of these models tend to exhibit higher brain alignment (Toneva & Wehbe 2019 Caucheteux & King. 2020). (ii) Unlike the MPL model, both the InstructBIIIP and IDEFICS models show minimal impact of later layer alignment on high-level visual regions.
6.3 PARTITIONING EXPLAINED VARIANCE BETWEEN TASK-SPECIIG INSTRUCTIONS
While the previous analysis reveals that not all instructions result in higher brain alignment across visual ROIs, we disentangle representations of task-specific instructions to measure the overlap in brain variance explained by MLLMs. To accomplish this we variance partitioning approach discussed in Section 5 Using this approach, we measure the brain response variance explained by pairs of instruction representations, separating their unique and overlapping contributions. Variance partitioning quickly becomes intractable as the number of feature spaces increases. Thus we restrict our analysis to pairwise comparisons that involve layers with max normalized brain alignment.
Shared variance across task task-specific instructions. Fig. 6 presents the shared variance between task prompts for the InstructBLIP model, averaged across all subjects. We observe the following from this figure: (i) High Overlap of Image Captioning (IC) with other prompts: The IC instruction exhibits high degree of shared information with prompts VQ1, VQ2. CR, and IU1. The promi- nent brain alignment of IC across multiple prompts suggests that it acts as the general or umbrella category for the other instruction prompts. This intuitive since order to generate an image cap- tion, the responses to other instructions are automatically taken into account. This broad relevance may account for its ability to capture various visual elements that are crucial across different tasks, driving consistent brain alignment. (ii) Lower shared variance with IU2 and SR: Instructions like IU2 and SR demonstrate consistently lower shared variance with other prompts, particularly VQ and IC. This indicates that these instructions may elicit distinct neural responses or emphasize specific aspects of visual stimuli not represented by the broader visual tasks linked to VQ or IC. Further, to verify these results, we plotted the percentage overlap across image categories and show it in Fig.
15 Based on that, we see that the instruction IU2 which is concerned with "food" category would naturally have low shared variance with other instructions, since the overlap of "food" category is much lower with other categories, except for "kitchen", "furniture", and "person". And, we see sim- ilar trends for the instruction SR, which explains low shared variance, by looking at the "outdoor" and "indoor" image categories.
Overall, these observations underscore why IC, IU2, and VQ2 have task-specific brain alignments, exhibiting unique aspects of brain responses to visual regions, as illustrated in Fig.
ROT Analysis: Shared and Unique variance across task-specific instructions.
Fig. 7 presents the unique and shared variance between task prompts-Image Captioning (IC) and Image Understanding (IU2)-for the InstructBLIP model, focusing on representative subject-1. 1. We present similar analysis between Image Captioning (IC) and Visual Question Answering 1 (VQI) in Appendix Fig. 14] From Fig. 7 we observe the following: (i) Between IC and IU2, there is no unique variance for IU2 in the EBA region, while IC retains some unique variance. Additionally, other high-level visual regions show a similar percentage of unique and shared variance for both
IC and IU2 instructions. (ii) In contrast to IC and IU2, between IC and VQ1, there is no unique

---

Published as a conference paper at ICLR 2025
Unique Variance-IC
Shared Variance-1C vs. IU2
Unique Variance-IU2
Figure 7: Unique & shared variances between pairs of task-specific instructions for the InstructBLIP model, focusing on representative subject-1. Image Captioning (IC) vs. Image Understanding (IU2).
Fig. 14 in Appendix Oopendix[shooee shows similar plots for IC vs. Visual Question Answering 1 (VQ1).
variance in the high-level visual region EBA, as this region is explained by shared information be- tween the models. (iii) In other high-level visual regions, a portion of the variance is unique to each model. though the majority is explained by shared variance. Overall, these findings highlight the role of shared neural processes across task-specific instructions in high-level visual regions while also demonstrating that different task instructions gions. However, the fact that the majority of variance is shared between instructions suggests room improvement in the MLLM models. Enhancing the models' ability to capture more task-specific information could lead to greater precision in predicting brain responses and better differentiation between various types of instructions. More detailed shared and unique variance analysis across task-specific instructions for whole visual cortex and ROI level are reported in Appendix L Our findings demonstrate that shared variance increases from early visual to higher visual areas, reflect- ing the hierarchical nature of visual processing.
drive distinct neural responses in specific
7 DISCUSSION AND CONCLUSION
Using instruction-tuned representations from MLLMs for various instructions, evaluated how well these representations predict fMRI brain activity when participants viewed naturalistic image stimuli. Additionally, we compared different MILMs' representations, assessing their alignment with each instruction across five visual brain regions. We show that MILMS exhibit significantly better brain alignment than vision-only models and perform comparably multimodal models.
Our analysis of instruction-tuned MLLMs and their brain aligmment reveals several key conclu- sions: (1) The effectiveness of task-specific instructions in predicting visual brain activity across different regions reveals that, although all three MLLMs generate task-specifi output tokens based on instructions, not all instructions lead to increased brain aligmment across all regions, as expected. Specifically, certain instructions (IC, VQ2, and IUI) may be more effective than others in encoding brain activity. This suggests that these instructions might serve as general or umbrella categories for other instruction prompts. This is further explained through a variance partitioning approach. (2) To uncover which brain regions are responsible for visual concept understanding, we examine how different instructions from MLLMs capture various visual concepts, such counts and recognition, as well as other concepts like color, positional understanding, and general scene understanding. We find that while MLLMs effectively capture count-related and recognition-relatee related visual concepts across instructions, they exhibit similar brain alignment patterns for other concepts. (3) By employ- ing variance partitioning approach, we measure the brain response variance explained by pairs of instruction representations, distinguishing between their unique and overlapping contributions. We find that most of the variance shared across instructions, with a high overlap between Image Captioning (IC) and other prompts but lower overlap with image understanding and scene recogni- tion. These results highlight potential area for improvement in MILMs, particularly in achieving better differentiation between various types of instructions. (4) The layer-wise representations from MLLMS correlate with visual functional localizers. In examining layer-wise trends in brain align- ment, we find that both InstructBLIP and IDEFICS demonstrate better brain alignment in middle layers for higher visual regions, while later layers align more with early visual regions. In contrast, mPLUG-Ow/ achieves higher brain alignment in the later layers for both high-level and early visual brain regions. This highlights the differences in information processing across the layers of MLLMs. We make the code publicly availablel] Lastly, we discuss limitations of our work in Appendix N
10

---

Published as a conference paper at ICLR 2025
8 ETHICS STATEMENT
We did not create any new neural recordings data as part of this work. We used the NSD dataset which is publicly available without any restrictions. NSD dataset can be downloaded from https:
//haturalscenessataaa
org/ Please read their terms of use for more details.
We do not foresee any harmful uses of this technology.
REFERENCES
Marah Abdin, Sam Ade Jacobs, Ammar Ahmad Awan, Jyoti Aneja, Ahmed Awadallah, Hany Awadalla, Nguyen Bach, Amit Bahree, Arash Bakhtiari, Harkirat Behl, et al. Phi-3 technical re- port: A highly capable language model locally on your phone. arXiv preprint arXiv:2404. 14219, August 2024.
Emily J Allen, Ghislain St-Yves, Yihan Wu, Jesse L Breedlove, Jacob S Prince, Logan T Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, et al. A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence. Nature Neuroscience, 25(1):116-11) 126,, 2022.
Khai Loong Aw and Mariya Toneva. Training language models to summarize narratives improves brain alignment. In The Eleventh International Conference on Learning Representations, 2023.
Khai Loong Aw, Syrielle Montariol, Badr AlKhamissi, Martin Schrimpf, and Antoine Bosselut. Instruction-tune -tninn aligns Ilms to the human brain. arXiv preprint arXiv:2312.00555, 2023.
Charlotte Caucheteux and Jean-Rémi King. Language processing in brains and deep neural net- works: computational convergence and its limits. Nature Communications Biology, 2020.
Colin Conwell, Jacob S Prince, Kendrick N Kay, George A Alvarez, and Talia Konkle. What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation brains and machines? bioRxiv, pp. 2022-03 03, 2022.
Wenliang Dai, Junnan Li, Dongxu Li, Anthony Meng Huat Tiong, Junqi Zhao, Weisheng Wang, Boyang Li, Pascale Fung, and Steven Hoi. Instructblip: Towards general-purpose vision-language models with instruction tuning. Advances in Neural Information Processing Systems, 2023.
Wendy de Heer, Alexander G Huth, Thomas Griffiths, Jack L Gallant, and Frédéric E Theunis- sen. The hierarchical cortical organization of human speech processing. Journal of Neuroscience, 37(27):6539 6557, 2017.
Fatma Deniz, Anwar O of semantic information across human cerebral cortex during listening versus reading is invariant to stimulus modality. Journal of Neuroscience, 2019.
Alexander G Huth, and Jack I. Gallant. The representation
Adrien Doerig, Tim C Kietzmann, Emily Allen, Yihan Wu, Thomas Naselaris, Kendrick Kay, and Ian Charest. Semantic scene descriptions objective of human vision. arXiv preprint arXiv:2209.11733 2022.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An im- age is worth 16x16 words: Transformers for image recognition at scale. International Confer- ence on Learning Representations, 2020.
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models. arXiv preprint orXiv:2407.2 1783, 2024.
Isabel Gauthier, Thomas W James, Kim M Curby, and Michael J Tarr. The influence of conceptual knowledge on visual discrimination. Cognitive Neuropsychology, 2073-5) 22tp://hhoooooo00000000000022220 2003.
https://cvmlab. s1 ite. htteeeaaaepp/////eeeeee httei/eeeewooTTeemmmmeee rese ant-mmod-n--onn---ii

---

Published as a conference paper at ICLR 2025
Ariel Goldstein, Zaid Zada, Eliav Buchnik, Mariano Schain, Amy Price, Bobbi Aubrey, Samuel A Nastase, Amir Feder, Dotan Emanuel, Alon Cohen, et al. Shared computational principles for language processing in humans and deep language models. Nature Neuroscience, 25(3):369-380, 2022.
Shailee Jain and Alexander G Huth. Incorporating context into language encoding models for fmri. In NIPS, pp. 6629-6638, 2018.
Albert Q Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, et al. Mistral 7b. arXiv preprini arXiv:2310.06822 2023.
Hugo Laurençon, Lucile Saulnier, Léo Tronchon, Stas Bekman, Amanpreet Singh, Anton Lozhkov, Thomas Wang. Siddharth Karamcheti, Alexander Rush, Douwe Kiela, et al. Obelics: An open web-scale filtered dataset of interleaved image-text documents. Advances in Neural Information Processing Systems, (1671683-77770 -1702, 2023.
Amanda LeBel, Shailee Jain, and Alexander G Huth. Voxelwise encoding models show that cere- bellar language representations are highly conceptual. Journal of Neuroscience, 41(50):10341- 10355, 2021.
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- image pre-training with frozen image encoders and large language models. In International Conference Machine Learning, pp. 19730-19742. PMLR, 2023.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jae Lee. Visual instruction tuning, Advances in Neural Information Processing Systems, 36, 2024.
Nikos K Logothetis. What we can do and what we cannot do with fmri. Nature, 453(71977:886--87,, 2008.
Khai Loong Aw, Syrielle Montariol, Badr AlKhamissi, Martin Schrimpf, and Antoine Bosselut. Instruction-tuning aligns Ilms to the human brain. First Conference Language Modeling, 2024.
Yuko Nakagi, Takuya Matsuyama, Naoko Koide-Majima, Hiroto Yamaguchi, Rieko Kubo, Shinji Nishimoto, and Yu Takagi. The brain tells story: Unveiling distinct representations of semantic content in speech, objects, and stories in the human brain with large language models. bioRxiv, pp. 2024-02, 2024.
Subba Reddy Oota, Jashn Arora, Veeral Agarwal, Mounika Marreddy. Manish Gupta, and Bapi Surampudi. Neural language taskonomy: Which nlp tasks are the most predictive of fmri brain activity? In Proceedings the 2022 Conference the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 3220-3237, 2022a.
Subba Reddy Oota, Jashn Arora, Vijay Rowtula, Manish Gupta, and Raju S Bapi. Visio-linguistic brain encoding. In COLING, pp. 116-133, 2022b.
Subba Reddy Oota, Agarwal Vecral, Marreddy Mounika, Gupta Manish, and Raju Surampudi Bapi. Speech taskonomy: Which speech tasks are the most predictive of fmri brain activity? In 24th INTERSPEECH Conference, 2023.
Subba Reddy Oota, Emin Çelik, Fatma Deniz, and Mariya Toneva. Speech language models lack important brain-relevant semantics. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pupers), pp. 8503-8528, 2024a.
Subba Reddy Oota, Zijiao Chen, Manish Gupta, Bapi Raju Surampudi, Gaël Jobard, Frédéric Alexandre, and Xavier Hinaut. Deep neural networks and brain alignment: Brain encoding and decoding (survey). Transactions on Machine Learning Research Journal, 2024b.
12

---

Published as a conference paper at ICLR 2025
Subba Reddy Oota, Manish Gupta, and Mariya Toneva. Joint processing of linguistic properties in brains and language models. Advances in Neural Information Processing Systems, 36, 2024c.
Sara F Popham, Alexander G Huth, Natalia Y Bilenko, Fatma Deniz, James Š Gao, Anwar O Nunez- Elizalde, and Jack L Gallant. Visual and linguistic semantic representations are aligned at the border of human visual cortex. Nature Neuroscience, 24(11):162811633 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision, 2021a. URL lttpss://axxis
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. Image, 2:T2, 2021b.
Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A Hosseini, Nancy Kan- wisher, Joshua Tenenbaum, and Evelina Fedorenko. The neural architecture language: In- tegrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, 2021.
Paul Scotti, Atmadeep Banerjee, Jimmie Goode, Stepan Shabalin, Alex Nguyen, Aidan Dempster, Nathalie Verlinde, Elad Yundler, David Weisherg, Kenneth Norman, et al. Reconstructing the mind's eye: fmri-to-image with contrastive learning and diffusion priors. Advances in Neural Information Processing Systems, 36, 2024a.
Paul Steven Scotti, Mihir Tripathy, Cesar Torrico, Reese Kneeland, Tong Chen, Ashutosh Narang, Charan Santhirasegaran, Jonathan Xu, Thomas Naselaris, Kenneth A Norman, et al. Mindeye2: Shared-subject models enable fmri-to-image with hour of data. Forty-first International Conference Machine Learning, 2024b.
Jingyuan Sun and which better account for brain language representations? In Proceedings of the Thirty-Second International Joint Conference on Artificial Intelligence, pp. 5197-5205, 2023.
Moens. Fine-tuned vs. prompt-tuned supervised representations:
Jingyuan Sun, Xiaohan Zhang, and human brain and artificial supervised representations of language. In ECA/ 2023, pp. 2258-2265. IOS Press, 2023.
Moens. Tuning in neural encoding: Linking
Jerry Tang, Meng Du, Vy Vo, Vasudev Lal, and Alexander Huth. Brain encoding models based on multimodal transformers can transfer across language and vision. Advances in Neural Information Processing Systems, 36, 2024.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B Hashimoto. Stanford alpaca: An mnollwcttoon-ollooinng llama model, 2023.
Mariya Toneva and Leila Wehbe. Interpreting and improving natural-language processing (in ma- chines) with natural maaguage-processing (in the brain). Advances in Neural Information Process- ing Systems, 32, 2019.
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, al. Llama: Open and efficient foundation language models. arXiv preprint 2302 2023.
Greta Tuckute, Jenelle Feather, Dana Boebinger, and Josh H McDermott. Many but not all deep neural network audio models capture brain responses and exhibit correspondence between model stages and brain regions. Plos Biology, 21(1)):3300366, 2023.
Aditya Vaidya, Shailee Jain, and Alexander Huth. Self-supervised models of audio effectively plain human cortical responses to speech. International Conference on Machine Learning, pp- 2 21927-21944. PMLR, 2022.
13

---

Published as a conference paper at ICLR 2025
Aria Wang, Michael Tarr, and Leila Wehbe. Neural taskonomy: Inferring the similarity of task- derived representations from brain activity. Advances Neural Information Processing Systems, 32:15501-15511, 2019.
Aria Y Wang, Kendrick Kay, Thomas Naselaris, Michael J Tarr, and Leila Wehbe. Natural language supervision with large and diverse dataset builds better models of human high-level visual cor- tex. BioRxiv, pp. 2022-09, 2022.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, et al. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, 2020.
Zhiyang Xu, Ying Shen, and Lifu Huang. Multinnstruct: Improving multi-modal zero-shot learning via instruction tuning. In Proceedings of the 61st Annual Meeting of the Association for Compu- tational Linguistics (Volume 1: Long Pupers), pp. 11445-11465, 2023.
Qinghao Ye, Haiyang Xu, Guohai Xu, Jiabo Ye, Ming Yan, Yiyang Zhou, Junyang Wang, Anwen Hu, Pengcheng Shi, Yaya Shi, et al. mplug-owl: Modularization empowers large language models with multimodality. arXiv preprint arXiv:2304. 14178, 2023.
Wanrong Zhu, Jack Hessel, Anas Awadalla, Samir Yitzhak Gadre, Jesse Dodge, Alex Fang, Young- jae Yu, Ludwig Schmidt, William Yang Wang, and Yejin Choi. Multimodal c4: An open, billion- scale corpus of images interleaved with text. Advances in Neural Information Processing Systems, 36, 2024.
14

---

Published as a conference paper at ICLR 2025
A
OVERVIEW OF APPENDIX SECTIONS
Section
Visual functional localizers
Section
Cross-subject brain predictivity
Section
Details of MLLMs with training details and their parameters
Section
Model generated outputs across instructions
Section
Normalized brain alignment: Visual functional localizers
Section
Brain Maps for Task-specific Instructions
Section
Brain Maps for different MLLM Layers
Section
1
ROI Analysis: Shared and Unique variance across task-specific instructions.
Section J
Normalized Brain Alignment across different image categories.
Section
K
Comparison of Instruction-tuned MLLMs, Non-Instrruction-tuned MLLMs and
text-based LLMs.
Section Whole Visual Cortex and ROI Analysis: Shared and Unique variance across task-specific instructions.
Section
M
Image only Instruction only input to the instruction- tuned MLLM.
Section
N
Limitations.
B VISUAL FUNCTIONAL LOCALIZERS
The NSD data covers five brain regions of interest (ROIs) in the human brain with the following sub- divisions: early visual (pRF- Visual ROIs) and high-level visual (floc-bodies, floc-words, floc-faces and floc-places) (Allen et al. 2022).
floc-bodies collection of manually drawn ROIs based on results of the floc experi- ment. These ROIs consist of EBA, FBA-1, FBA-2, and mTL--bodies ("mid temporal lobe bodies").
floc-words is a collection manually drawn ROIs based on results the floc experi- ment. These ROIs consist of OWFA, VWFA-1, VWFA-2, mfs-words ("mid fusiform sulcus words"), and mi'.-words ("mid temporal lobe words").
floc-faces a collection of manually drawn ROIs based on results of the floc experiment. These ROIs consist of OFA, FFA-1, FFA-2, mTL-faces ("mid temporal lobe faces"), and aTL-faces ("anterior temporal lobe faces").
floc-places is collection of manually drawn ROIs based on results of the floc experiment. These ROIs consist of OPA, PPA, and RSC.
pRF- Visual ROIs (Retinotopic Early Visual) is collection of manually drawn ROIs based on results of the prf experiment. These ROIs consist of VIv, Vid, V2v, V2d, V3v, V3d, and hV4
C CROSS-SUBJECT BRAIN PREDICTIVITY
We estimate cross-subject prediction accuracy for the NSD dataset and present the average accuracy across voxels for each subject in Fig. g The figure suggests that the cross-subject prediction accu- racy is consistent across subjects, indicating that all subjects share a similar amount of explainable variance.
D DETAILS OF MLLMS WITH TRAINING DETAILS AND THEIR PARAMETERS
We provide details about model parameters, training dataset details, training procedure details and task-specific instructions for instruction tuning of these models in Table
15

---

Published as a conference paper at ICLR 2025
Cross- sus--suueeeet..eee predictivity
0.2
0.1
Figure Cross-subject prediction accuracy for each subject of NSD dataset.
InstructBLIP Dai et al., 2023) is a vision-language instruction-tuned model built upon the pre- trained BLIP-2 model (Li et al. 2023). It leverages diverse set of instruction data (26 different datasets) to train a MI..M, which comprises an image encoder, large language model (LL.M,, and Query Transformer (Q-Former) that serves as bridge between the two. We utilize the instructblip- vicuna-7b version, which consists of 32 layers and produces 4096-dimensional representations.
mPLUG-Owl (Ye et al. 2023) is MLLM designed to perceive and integrate multiple modalities (visual and language) while considering visual context and information to generate corresponding outputs. The model is trained on language modeling task, which involves learning to generate subsequent tokens based on the preceding context. We utilize the which consists of 32 layers and -096-dimensional representations.
version,
IDEFICS Laurençon et al. 2023) is an MLLM based on Flamingo (huu et al. 2024), which accepts arbitrary sequences of image and text inputs and generates text tokens. We utilize the idefics-9b- instruct version (the model obtained by further training IDEFICS on supervised fine-tuning and instruction fine-tuning datasets), which consists of 32 layers and 4096-dimensional representations.
Table 2: MLLM Training Details and Parameters
Model
Architecture
Training Dataset
Task-Specifle Instruc-
Parameters Training Frocedure
lions
InstrucBLIPP
Large
Image captioning, visual question answering, im- ago understanding
Supervised image. text data, followed by instruction-tuning
image-text instruction-
pairs;
following datasets
m mPLUG-OWL
Vision-language transtormer
Image-text
pairs:
Visual reasoning, image
14B
Masked language mod- cling. next-werd predic- tion. visual grounding. instruction-tuning
multimodal datasets
captioning. genera-
tion
IDEFICS
Multimodal transformer
COCO.
Vi- Image captioning,
10B
Contrastive learning image-text pairs, task- specilic line-tuning
sual
Genome:
sual questton answering.
luning
scene understanding
Implementation details for reproducibility. All feature extraction experiments were conducted on machine equipped with NVIDIA A100 GPU with 80 GB of GPU RAM, partitioned into two devices of 40 GB each. The voxelwise encoding models were trained on NVIDIA GeForce RTX 3050 GPU with 4GB of GPU RAM. We used bootstrap regeeseegession with the following parameters: MSE loss function; L2-decay (X) varied from 10-1 to 10s; the best À was chosen by tuning on validation data that comprised randomly chosen 10% subset from the train set used only for thyper-paramedd tuning.
E MODEL GENERATED OUTPUTS ACROSS INSTRUCTIONS
Tables and 6 show model generated outputs for sample image from the NSD datasct using InstructBLIP mPLUG-Owl, IDEFICS and BLIP-2 models respectively.
16

---

Published as a conference paper at ICLR 2025
(a) Subject-01
(b) Subject-02
(c) Subject-05
(d) Subject-07
Figure 9: Contrast of estimated cross-subject prediction accuracy for the participants for the NSD dataset.
17

---

Published as a conference paper at ICLR 2025
Table 3: InstructBLLIP generated outputs for sample image from the NSD dataset.
Instruction
Model Oulput
Image Captioning
pulling train station
The image features a blue and yellow train pulling train station.
Question Ans wering
are people this image.
The loreground of the imuge teatures train, while the buckground consists of train station.
Visual Relationship
the image, the train moving through train station and interacting with a person who has their back turned it.
Commonsense Reasoning
The image shows train station.
The most dominant color in the image blue.
Image Understanding
There
lood Hiems visibll the imuge.
There are two animals the image.
Socne Recognition
image features a yellow color.
Table 4: mPLUG-OWl generated outputs for sample image from the NSD dataset.
Instruction
Model Output
Image Captioning
group of kids is riding their skateboards street.
The image Jeatures a young boy weating a red shirt and a black while riding a skatebourd a skate park. There several other children around, watching engaging with the skulebourder their own skatebourds.
Visual Question Answoring
image, including boy appears skateboarding another boy who watching the skateboarder, possibly learning to skateboard or observing the skill
Yes, there
people
In the foreground of the image, there is group of young people. including boy and few other kids riding skatebourds on dirt or park with skateboard ramp. In the background, car cun seen
parked the skatehoard
Visual Relationship
In the image, a group of young people are playing on skateboards and riding bikes in city setting. person bike
skatebourler
Commonsense Reasoning
The image shows playgrunnd environment, specifically small skatepark with few people engaged in various activities such as rollerblading, skateboarding, and biking.
most dominant color image
Image Understanding
In the image, few food items are visible, including of lee cream and bottle of water.
There are two animals the image- a dog cat.
Scene Recognition
The scone in the image showcases natural outdoor skate park, where several young skatchoarders enjoying their time on the ramps and rails.
F NORMALIZED BRAIN ALIGNMENT: VISUAL FUNCTIONAL. LOCALIZERS
Fig. I0 shows ROI-based normalized brain alignment computed by averaging across participants, layers, and voxels. The figure shows the alignment values for FLOC-BODIES, FLOC-FACES and FLOC-WORDS regions.
18

---

Published as a conference paper at ICLR 2025
Table 5: IDEFICS generated outputs for sample image from the NSD dataset.
Instruction
Model Ouiput
Image Captioning
In crowded outdoor marketplace, the sun beats down who standing beneath a large bunch
of bright yellow hananas.
indeor market area, surrounded by large number ripe several people in the image, man, a blue jacket.
This image features
hanging from a structure.
seen in backgrand wearing There is no interaction between animate and inanimate objects.
Visual Question Answering
the foreground, there are rows of hamanas hanging in a marketplace, In the hackground, people are walking around the area.
Visuul Relationship
Commonnsense Reasoning
The image shows market enviroament with man standing next to hanging bunches bananas.
The dominant color of image appears yellow, there lot of yellow image, possibly from the color of bananas.
Image Understanding
image, Bananas are visible.
There are animals in the image. In the image, there is section of banana stand with bananas displayed on the wall.
Seene Recognition
Table 6: BLIP-2 generated outputs for sample image from the NSD dataset.
Instruction
Model Output
Image Captioning
passenger pulling train station
A train
Question Answering
blue train trann track with platform the side It's physical one
Visual Relationship
Commonsense Reasoning
Mation
Blue
Image Understanding
Sand
No animals
Scene Recognition
This can he picture, a video photograph
We conducted a non-par non-maraettric one-way ANOVA test to analyze the normalized brain alignment differences between early and higher visual ROIS for each instruction-tun -uuned MLLM across four subjects.
The results for InstructBIIIP Model are follows. The one-way ANOVA test revealed that higher visual ROIs have significantly higher normalized brain alignment than early visual ROIS, with p-value of 0.008 and an F-statistic of 14.60.
Pairwise Comparisons Between Early Visual and Higher Visual ROIs: We further performed ANOVA tests between early visual ROIs and each specific higher visual ROI. The results indicate that for most higher visual ROIs, the brain alignment significantly better compared to early vi-
19

---

Published as a conference paper at ICLR 2025
Random
Normalizd train aigurrent
0.9
Nornalbed brain nomment
0.9
R.A
0.7
3007
0.6
9.
P.3
2.4
D4
0.4
6.2
0.2
0.2
6.4
D.
PLOC-BODIES
FLOC-FACES
FLOC-WORDS
Figure 10: ROI-based normalized brain alignment was computed by averaging across participants, layers, and voxels. Blue: Average across random initialization of the MLLMs. Note that CLIP-text model uses golden oracle captions while instruct models use predicted model generations. indi- cates cases where MLLM embeddings are statistically significantly better than randomly initialized models, i.e., P.. 0.05. A indicates cases where MLLMs are significantly better than unimodal vision models (ViT-HH, i.e., PS 0.05.
sual ROIs: Early Visual vs. FLOC-PLACES (p-value: 0.006, F-statistic: 41.45), Early Visual vs. FLOC-Bodies (p-value: 0.001, F-statistic: 32.91), Early Visual vs. FLOC-Faces (p-value: 0,006, F-statistic: 41.02), Early Visual vs. FLOC-W Words (p-value: 0.14 (not statistically significant), F-
statistic: 2.83). These results quantitatively confirm that multimodal models such as InstructBLIP achieve significantly better alignment higher visual ROIs than in early visual ROIs
We performed similar analysis to include all instruction -undet MLLMs and conducted one-way ANOVA test to compare normalized brain alignment between early visual and higher visual ROIs. The one-way ANOVA test revealed that higher visual ROIs have significantly higher normalized brain alignment than early visual ROIs, with a p-value 0.009 and an F-statistic of 13.85.
G BRAIN MAPS FOR TASK PASK-SPECIFIC INSTRUCTIONS
The Fig. 11 displays brain maps for the InstructBLIP. and mPLUG-OWL for Subject 1, where the voxel color codes are projected onto the flattened cortical surface the representative subject.
The Fig. 12 displays brain maps for the InstructBLIP model for Subject 1, with normalized brain alignment voxels are projected onto the flattened cortical surface of the representative subject for two concept groups: Color, Position and Scene understanding.
H BRAIN MAPS FOR DIFFERENT MLLM LAYERS
Fig. 13 presents brain maps for the InstructBLIP, mPLUG-Owl and IDEFICS, where the voxels with their corresponding color codes are projected onto the flattened cortical surface of the representative subjects (Subjects and 7).
ROI ANALYSIS: SHARED AND UNIQUE VARIANCE ACROSS TASK-SPECIFIC INSTRUCTIONS
Fig. presents the unique and shared variance between task prompts-Imaee Image Captioning (IC) and Visual Question Answering 1 (VQI)-for for the InstructBLIP model. focusing on representative su From Fig. 14 we observe the following: (i) Between IC and VQ1, there is no unique variance in the high-level visual region EBA, as this region is explained by shared information between the models. (ii) other high-level visual regions, a portion of the variance is unique to each model, though the majority is explained by shared variance. Overall, these findings highlight the role of shared neural processes across task-specific instructions high-level visual regions while also demonstrating that different task instructions drive distinct neural responses in specific regions. However, the fact that the majority of variance is shared between instructions suggests room for improvement in the MILM models. Enhancing the models' ability to capture more task-specific
20

---

Published as a conference paper at ICLR 2025
S1: IDEFICS
PSCC
S2: InstructBIIP
$2: mPLUG-Owl
S3: InstructBLIP
S3: mPLUG-Owl
84: InstructBLIP
84: mPLUG-OWl
Figure 11: Each voxel is color coded with the instruction (out of 10) that led to the highest nor- malized brain alignment. The color bar highlights color codes for each instruction. The voxels projected onto the flattened cortical surface of a representative subject (subject S1, S2, S5 and S7) for three MILMS.
21

---

Published as a conference paper at ICLR 2025
ORA
Cony
Eeryy
Earty Vir
Color / Texture
Position understanding
Scene understanding
Figure 12: The visual concept specific voxels projected onto the flattened cortical surface of representative subject-1.
Layer depth
1123 4 9 10012234411777333012233333333333
InstructBLIP
S2: mPLUG-Owl
S2: IDEFICS
$3: InstructBLIP
$3: mPLUG-Owl
S3: IDEFICS
$4: InstructBLIP
S4: mPLUG-Owl
S4: IDEFICS
Figure 13: Each voxel is color coded with the MILLM layer number (out of 33) that led to the highest normalized brain alignment. The color bar highlights color codes for each layer. The voxels projected onto the flattened cortical surface of representative subject (S2, S5 and S7) for three MLLMs.
information could lead to greater precision in predicting brain responses and better differentiation between various types of instructions.
22

---

Published as a conference paper at ICLR 2025
Unique Variance-IC
Shared Variance-IC vs. VQ1
Unique Variance-VQ1
Figure 14: Unique and shared variances between pairs of task-specific instructions for the Instruct- BLIP model, focusing on representative subject-1. Image Captioning (IC) vs. Visual Question Answering (VQ1).
J NORMALIZED BRAIN ALIGNMENT ACROSS DIFFERENT IMAGE CATEGORIES
The NSD dataset contains images from 12 different categories. Each image can be labeled with multiple categories. Fig. [15]shows the percentage of overlap between pair of categories. Some pairs have very high overlap values as expected like (person, sports), (food, kitchen), (furniture, kitchen), (person, vehicle), etc.
To understand the effectiveness of task-specific instructions to brain alignment across various im- age categories, we performed a category-wise analysis, where we computed the normalized brain alignment for voxels in each category by averaging across all task instructions, and across all the MLLMs. Fig. To]shows the normalized brain alignment for five visual functional localizers: FLOC- BODIES, FLOC-PLACES, FLOC-FACES, FLOC-WORDE and PiF-Visual ROIs.
Do the MILLM representations demonstrate better brain predictivity for certain categories of image stimuli?
To understand the effectiveness of task saeciipc instructions brain alignment across various im- age categories, we performed category-wise analysis, where we computed the normalized brain alignment for voxels in each category by averaging across all task instructions, and across the MLLMs. Fig. 16 shows the normalized brain alignment for FLOC-BODIES and FLOC-PLACES ROIs. From this, we observe the following: (i) MLLMs exhibit higher normalized brain alignment for person category test. images in the FLOC-BODDES region, suggesting that instruction-speccfic representations effectively capture body-related information, resulting in higher alignment. We also observe higher alignment in the furniture category, likely due its 15% overlap with person-related images, as shown in Fig. [5] (ii) For FLOC-PLACES ROI, similar to FLOC-BODIES both furni- ture and kitchen categories test images have higher alignment, as these categories share overlapping image content (30%). (iii) Although the appliance category has partial overlap with furniture and kitchen, MLLMs display lowest normalized brain alignment for this category. This suggests that the appliance category may contain unique features not as well captured by the MILM's repre- sentations, it may lack the strong shared features that drive high alignment other categories. These trends are consistent across other visual ROIs, such as FLOC-WORDS. FLOC-FACES, and PRF-Visual ROIs, as shown in Fig. [6
Overall, the analysis shows that MLLMs are most effective at aligning with brain activity when both task-specific and shared category features are present, suggesting that these features play a crucial role in brain alignment. However, in categories with fewer shared visual elements (c.g., appliances), alignment is weaker, indicating that these models may have difficulty fully capturing certain types of visual information.
23

---

Published as a conference paper at ICLR 2025
Percentage overlap across image categories
100
accessory
100.00
1.32
5.97
appliance
1.32
100.00
3.10
11.03
0.26
1.32
3.20
5.45
19.63
8.69
0.20
0.08
electronic
7.96
3.10
100 .0
16.63
2.30
3.82
9.94
3.15
12.13
14.34
2.09
0.72
fumiture
9.37
11.83
16.63
100.00
3.55
6.67
15.39
19. 06.06
30.33
26
3.69
3.77
vehicle
animal
4.98
1.32
3.82
6.67
5.75
100.00
8.73
1.40
3.26
3.74
3.21
2.54
person
22.59
3.20
9.94
15.39
23.57
8.73
100.00
8.43
11.74
7.89
11:18
36.49
food
3:29
5.45
19.06
1.55
8.43
3.76
kitchen
30.53
indoor
5.92
8.69
14.34
19.26
2.84
3.74
7.80
3.76
14.26
100
00
2.62
0.69
outdoor
10.1
0,20
2.09
3.69
18.96
3.21
11.18
0.87
1.99
2.62
100.00
4.68
sports
6.88
0.08
5.56
36.49
0.69
100.00
000000
accessory
appliance
electronic
furniture
vehicle.
animal
person
food
kitchen-
indoor
outdoor
sports
Figure 15: Percentage of overlap between pair of categories.
K COMPARISON OF INSTRUCTION- MLLMS, NON-INSTRUCTION-TUN MLLMS AND TEXT- BASED LLMS
To thoroughly explore the role of instruction tuning, we conducted two additional experiments: (i) Non-instruction-tuune MLLM (BLIP-2): We passed all task instructions input. (ii) Language Model (LLaMA-2-7B): We passed only captions as input without task instructions.
Generated output tokens from BLIP-2. Firstly, present the generated output tokens from the BLIP-2 model in 6 This table includes examples of the generated tokens for each task instruc- tion. We already provided predictions from InstructBLIP for the same image in Appendix Table From the examples in Table 6, we observe that the generated output tokens adhere more closely captioning instructions, regardless of the specific task instructions provided. The outputs often consist of simple responses, such as "Yes," "No," or color names, and lack detailed descriptions, In contrast, instruction-tunee MLLMs excel providing semantically rich and conceptually grounded descriptions, as shown in Table 3 demonstrating significant difference in their ability to follow task specifcc instructions effectively.
Comparison with normalized brain alignment scores across voxels. Second, we present the scatter plot in Fig. 17, which compares the performance of InstructionBLIP vs. BLIP-2 for brain predictivity. The plot includes all voxels across the visual cortex with normalized brain alignment scores, where the diagonal represents identical performance for both models.
Image Captioning Task (left): The histogram shows distribution of voxels deviating from the diagonal towards InstructBLIP, indicating that InstructBLIPP performs better. However, the deviation is more pronounced for voxels with normalized brain alignment scores 0.2.
Visual Relation Task (right): Similarly, the voxel distribution deviates significantly towards Instruct- BLIP, demonstrating its superior performance. The deviation is notably larger for this task compared to the image captioning task.
Whole Visual Cortex vs. ROI-level Analysis Third, we extended the analysis from Fig. of the main paper by computing normalized brain alignment for the following regions: (a) W Whole
Visual Cortex, (b) Early Visual Cortex (pRF-Vise Visuall and (c) High-level Visual Cortex (Bodies, Faces, Places, Words) The results the Fig. [8] demonstrate that the -om-instuuctinn-tet MLLM (BLIP-2) achieves brain alignment scores that marginally below those of the CLIP-Text model.
24

---

Published as a conference paper at ICLR 2025
accessory
furniture
person
Indoor
accessory
furniture
person
indoor
appliance
applience
vehicle
food
electronic
animal
kitchen
electronic
aniimal
kitchen
sports
Normalized brain alignment
Normalisn alzzalzzd brain aligmment
0.8
0.8
0.7
0.6
0.6
0.5
0.5
FLOC-BODIES
FLOC-PLACES
accessory
accessory
furniture
Indoor
appliance
vehicle
food
outdoor sports
appliance
vehicle
food
outdoor
electronic
animal
kitchen
electronic
animal
kitchen
sports
Normalized anlizeddd brain alignment
Normalized brain alignment
0.8
0.8
0.6
0.6
0.4
0.4
FLOC-FACES
FLOC-WORDS
accessory
fumiture
person
Indoor
appliance
electronic
animal
kitchen
Normalized brain alignment
0.8
0.7
0.5
PRF-VISUALROIS
Figure 16: Average (across MLLMs) normalized brain alignment across 12 categories of im- ages for two function localizers (FL.CC-F))) (FLOOCBBODESS FLOC-PLACESS FL.OC-FACES, FLOC-WORDS and pRF-Visiee Visual ROIs). Error bars indicate the standard error of the mean across participants.
In contrast, the LLaMA-2-78 2-BB model demonstrates performance closer the VIT-H model. This behavior is attributed to the fact that non-instruction-tumee models generate output tokens that adhere more closely to captioning instructions, regardless specific task instructions.
Key Findings
The performance boost in instraction-tuned MLLMs is primarily due to their ability to generate semantic descriptions and conceptually understand the elements of each scene, opposed to merely generating task- answers, in non-instruction-tumee MLL.Ms.
Representations from the LLaMA-2-7B model, which are based solely on captions, ex- hibit more semantic information but lack visual understanding details. Consequently, its performance aligns more closely with visual model representations (e.g., ViT).
Overall, Instruction-tumne MLLMS demonstrate superior visual understanding when provided with task-specific instructions, evidenced by their higher brain alignment. This reinforces the value of instruction tuning in enhancing multimodal understanding and alignment with human brain repre- sentations.
25

---

Published as a conference paper at ICLR 2025
ImageCaptioning Task
Visual Relationship Task
Non Intrrctionn turee MLLU: BUP.
0.8
0.8
0.6
0.6
0.4
10
number 10
number
0.
0.0
10
10
Instruction-tued MLLM: InstructBUPP
Instruction-tuned MILLM InstructBUiP
Figure 17: Comparison of MLLMs performance (Instruction-tune vs. instruction-tumee with 2D scatter plot. Normalized brain alignment of all voxels is represented in the plot. The diagonal corresponds to identical performance for both models. A distribution of voxels deviating from the diagonal towards InstructBLIP means that InstructBLIP is performing better than BLIP-2. Left plot is for Image Captioning task, and right plot is for Visual Relation task. Plot shows that InstructBLIP performs better than BLIP-2.
Bormeling leeelddd brein sligmment
0.9
Normalaed altgnmantt
cemment
L1AM66-277
0.5
0.7
0.6
0.6
0.0.5
24
0.1
0.1
WP
FLOC-BODGER
FACES
* Rendom Inttialtattion
Ineructii.1p
Random Initelization
IrstructBUPP
Random Init altnation
IncructSLIP
IDEFICS
MPLUG-OWL
IDEFICS
MPLUG OWL
IDEFICS
MPJUG-OWL
VITTH
CLIP-Text
VIT-I
CLIR-Toit
VITH
CLI-Test
aligemen
0.9
RI 99-3
11 11-66-77
Normaltzed brain allgmment
R/1P-2
11-M6-2-78
algemen
0.9
RJ 19-3
11866-2-78
0.8
0.0
0.5
0.7
4AA
3.5
0.5
9.4
0.4
0.4
0.3
0.3
Normeliey
0.3
0.
0.7
[cI FLDOIFLACES
FLOD WORDS
(b) pRF-ViuM RDN
Figure 18: Whole Visual Cortex and ROI-based normalized brain alignment was computed by av- eraging across participants, layers, and voxels. Blue: Average across random initialization of the MLLMs. Note that CLIP-text model and LLaMA models use golden oracle captions while instruct and non-instruct models use predicted model generations. indicates cases where MLLM embed- dings are statistically significantly better than randomly initialized models, i.e., ps 0.05. A indicates cases where MILMS significantly better than unimodal vision models (ViT-H), i.e., ps 0.05.
L WHOLE VISUAL CORTEX AND ROI ANALYSIS: SHARED AND UNIQUE VARIANCE ACROSS TASK-SPECIFIC INSTRUCTIONS
Figure shows Venn diagrams representing the shared and unique variance between the task and nine other task-based instructions (VQ1, VQ2, VR, CR, IU1, IU2, IU3, SR) across the whole visual cortex. We make the following observations from Figure 19 (1) The higher shared variance across most task pairs (e.g., IC & VQI, IC & CR), highlighting that tasks involving visual question answering and captioning share significant neural processing mechanisms in the visual cortex. (2) IC retains notable amount of unique variance in most comparisons, suggesting that certain neural processes underlying image captioning are specific to this task and not shared with other visual tasks. (3) Other tasks also retain unique variance, which might reflect the task-specific nature of
26

---

Published as a conference paper at ICLR 2025
these activities (e.g., VQ1-VQ3 could involve question-based reasoning, and IU tasks could involve higher-level image understanding.
Similarly, Figures 20 and 21 show Venn diagrams for the early visual and higher visual regions of interest (ROIs), depicting shared and unique variance across these regions. We make the follow- ing observations from Figures 20 and 21 (i) Unlike the whole visual cortex, tasks like IC and other task-specific instructions exhibit moderate shared variance the early visual cortex, while shared variance significantly higher in higher visual ROIs. This suggests that these tasks depend on similar low-level visual processing mechanisms in early visual areas. (ii) The IU1 instruction ("Describe the most dominant color the image") shows greater unique variance than IC in the early visual cortex, indicating that low-level color processing is specific to early visual areas, with MILMMs effectively capturing task-specific representations. In contrast, IC exhibits greater unique variance in the higher visual cortex compared to IU1, reflecting the task's reliance on higher-order visual processing.
Overall, these findings demonstrate that shared variance increases from early visual to higher visual areas, reflecting the hierarchical nature visual processing. Meanwhile, unique variance decreases in higher areas, as tasks rely more on integrated and shared representations. Tasks such as Visual Question Answering (VQ) and Visual Reasoning (VR) retain distinct processing features across both early and higher ROIs, underscoring their unique demands visual and cognitive processing. These results highlight the capacity of MILMs to distinguish task-specific representations and align closely with brain visual processing mechanisms.
0.186
0.446
0.192
VQI
IC
VQ2
VQ3
0.216
0,417
0.199
VR
0234
IU2
IUB
Figure 19: Whole Visual Cortex: Shared and Unique Variance explained between task instructions: Image Captioning (IC) and other task instructions. In each plot, Pink Area (Left Circle Intersec- tion) represents the unique variance explained by the IC task that is not shared with the correspond- ing task. Green Area (Right Circle Intersection) represents the unique variance explained by the corresponding task (e.g., VQI, CR, etc.) that is not shared with the IC task. Light Brown Inter- section (Overlap) represents the shared variance between the IC task and the corresponding task. It indicates the extent which both tasks explain overlapping neural variance in the visual cortex.
27

---

Figure 20: Early Visual Cortex: Shared and Unique Variance explained between task instructions: Image Captioning (IC) and other task instructions. In each plot, Pink Area (Left Circle Intersec- tion) represents the unique variance explained by the IC task that is not shared with the correspond- ing task. Green Area (Right Circle Intersection) represents the unique variance explained by the corresponding task (c.g., VQI, CR, etc.) that is not shared with the IC task. Light brown Intersection (Overlap) represents the shared variance between the IC task and the corresponding task. It indicates Published as a conference paper at ICLR 2025
0.22
0-252
0233
0.252
0.238
2,293
0.243
IC
vQ1
VQ3
0.231
0.325
6.226
0.254
5,295
0.284
VR
2.278
0264 0.254
0.22 3,272
0,252
IC
IU2
IC
SR
the extent to which both tasks explain overlapping neural variance in the visual cortex.
M IMAGE ONLY INSTRUCTION ONLY INPUT TO THE INSTRUCTION-TUNED MLLM
We performed two additional experiments to investigate the behavior of instruction-tune -tunedd MLLMs, as reported in Fig. 22 Image-Only Input with Empty Prompt: Here, the input to the instruction- tuned MILMM consists only of images, with empty string provided the instruction prompt. From this experiment, we observed that the embeddings generated by the instruction-tuneet MILM using only images and an empty prompt perform similar the CLIP-image embeddings. Our hy- pothesis is that providing an empty instruction effectively reduces the instruction behave similarly formance to the CLIP model. This supports the notion that the absence of an instruction prompt shifts the model's behavior towards more vision-centrie embedding generation. Instruction-One -nnly Input with Empty Image: In contrast to image only input, when only task instructions are provided with no image input, the embeddings perform below the randomly initialized baseline. This demon- strates that visual input is crucial for achieving meaningful brain alignment, MLLMs with visual input, such as "InstructBLIP" (vision+language) and "InstructBLIP No Prompt" (vision-only), sig- nificantly outperform the instruction-only baseline.
MLLM to vision-language model like CLIP, resulting in comparable brain encoding per-
Fig. 22 illustrates the normalized brain alignment across the whole visual cortex, comparing the Instruct model with and without prompt, without image, well the ViT-H and Cexp-Text models. This figure shows that InstructBLIP with prompt achieves higher normalized brain align- ment compared the model without a prompt. Furthermore, when passing single modality
28

---

Published as a conference paper at ICLR 2025
0.225
0.503
0.239
0.25
0.475
0.24
VQ3
0.275
3:454
0.245
0.299
0.549
0,201
9.256
0.461
0.242
VR
CR
IU1
0.308
8,409
5.273
0483
0261
IU2
SR
Figure 21: Higher Visual Cortex: Shared and Unique Variance explained between task instruc- tions: Image Captioning (IC) and other task instructions. In each plot, Pink Area (Left Circle Intersection) represents the unique variance explained by the IC task that is not shared with the cor- responding task. Green Area (Right Circle Intersection) represents the unique variance explained by the corresponding task (e.g., VQ1, CR, etc.) that is not shared with the IC task. Light brown In- tersection (Overlap) represents the shared variance between the IC task and the corresponding task. indicates extent to which both tasks explain overlapping neural variance the visual cortex.
(image) the InstructBLIP model, it retrieves relevant embeddings from the text modality within its aligned embedding space, similar to the behavior of the CLIP model. This is likely due to the model leverage
being pretrained with images and their associated task-specific instructions, enabling its multimodal alignment. However, when only task instructions are provided without an image in- put, the embeddings perform below randomly initialized baseline, emphasizing the critical role of visual input in achieving meaningful alignment. This occurs because the InstructBLIIP model specifically designed to integrate both visual and textual information. Without the image input, the model loses critical part of the visual context, severely affecting its ability to comprehend and generate accurate responses. In scenarios with both image and instruction inputs, the model leverage the visual context to better interpret and respond to the instructions. In contrast, the absence of an image deprives InstructBLIIP of this advantage, leading to significantly poorer per- formance compared conditions where both inputs are provided. Overall, these results highlights the dependency of MLLMs on visual input for robust performance and supports the importance of including both visual and task-specific instructions for achieving high brain alignment.
N LIMITATIONS
Apart from the generic limitations of fMRI such poor temporal resolution (compared to EEG or MEG) and the blood oxygen level dependent (BOLD) signal being an indirect measure of neural activity (Logothetis) 2008), specific limitation of the current study is that it relies the NSD
29

---

Published as a conference paper at ICLR 2025
Random Initialization
InstructBLIP
InstructBLIP No Prompt
InstructBLIP Image
CLIP-Text
Normalizen alized brain aligmmett
0.9
0.8
0.6
0.5
0.4
0.3
0.1
0
(a) Whole Visual Cortex
Figure 22:
Whole Visual Cortex: Comparison of Instruction-tuned MLLMs performance (Instruct- BLIP with and without prompt, without image).
dataset, where subjects passively viewed images. As result, the dataset may fully capture how brain activity aligns with task-specific instructions. Collecting brain recordings while subjects engage tasks guided by different instructions could provide more comprehensive evaluation of whether instruction-tune -unde MLLMs truly reflect visual information processing in response to natural instructions. It is also important note that while observed several spec-fie instructions leading to improved brain alignment between fMRI recordings and MLLMs, not all instructions were relevant brain alignment. This indicates that our analysis may not have fully captured range of task- spsciipecific instructions that are jointly processed by the brain. Future work could expand on our approach by incorporating additional task-specific instructions to better characterize the joint processing of information between the brain and instruction-uun tuned MLLMs.
Since the NSD dataset was collected while participants engaged in watching scenes, our work pri- marily focuses the visual areas of the brain, these most relevant for the stimuli and tasks studied. Additionally, leveraging the associated ROI mappings of the NSD dataset, we conducted analysis across the entire visual cortex. Regions outside the visual cortex, such the language network, prefrontal cortex, or auditory regions, may not exhibit strong alignment with visual fea- tures due their specialization in non-visual tasks, such as language processing, decision-makee making, or auditory perception. However, it would be an interesting direction to explore the alignment using instruction-tune multimodal large language models in scenarios where participants watch videos involving auditory, visual, and language information, enabling comprehensive whole- brain analy- sis.
30


Reviews:
[{"review_id": "yv0cdSzthC", "reviewer": "Reviewer_3L7C", "summary": "The paper studies brain-encoding fits of instructions-tuned MLLMs in comparison with traditional methods for designing multimodal embeddings such as via CLIP. To study this question, the paper feeds different types of instructions to the MLLMs in comparison to other baselines like CLIP or vision encoders. The paper finds that instruction-tuned MLLMs have much better fits with visual areas of the brain than vision embedding models and similar performance to CLIP. The authors break down the instructions by type to identify which type of instructions have stronger correlation with visual areas of the brain and do the same with visual concept groups. The authors also include a shared variance experiment that aims to characterize whether shared features were used for each instruction group that were relevant for encoding model performance. \n\nOverall, I liked this paper and would vote for acceptance outside of a few concerns. It answered a relevant question about instruction-tuned MLLMs that haven’t really been explored before.", "questions": "* The paper focuses on the visual areas of the brain. How would the results look if the entire brain was used? A small discussion would suffice. \n* How many parameters are ViT-H and CLIP? Are they comparable?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Novelty with instruction-tuned MLLMs: Overall, I haven’t seen too much work on brain encoding models with instruction-tuned MLLMs. This work is highly timely.\n* Well-designed and controlled experiment: The paper uses a controlled and well designed experiment for exploring brain fits.\n* Instruction breakdown: I really appreciated Figure 3 and 4. I thought it was pretty interesting to see how different instruction types fit responses in the visual cortex. This was a fairly novel result. \n* I also really liked Figure 6 which focused on shared variance. This was an interesting result that showed which instruction types had a higher degree of shared features which resulted in a higher degree of brain similarity.", "weaknesses": "* Comparisons: Overall, I think it would be better to include a baseline with a non-instruction-tuned MLLM that has the same architecture. For example, maybe BLIP-2 instead of InstructBLIP? This would have really explored the role of instruction tuning more thoroughly in comparison. BLIP-2 should be available off the shelf and I believe this would be a salient comparison. I would also be curious about how a language model of similar size would do. \n* Another concern here is that improvement over ViT-H could be due to an increase in parameters. See questions. \n* Nit: Could Figure 3 be sharpened? The text is quite blurry and difficult to read. \n\nIncorporation of some of these comparisons would help raise my score/confidence.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "lNGC6VYRlg", "reviewer": "Reviewer_WsrE", "summary": "The paper investigates the alignment of instruction-tuned multimodal Large Language Models (MLLMs) with brain activity, particularly focusing on how these models process visual information when prompted with natural language instructions. The study explores the brain alignment of MLLMs compared to unimodal and non-instruction-tuned multimodal models and assesses the effectiveness of various task-specific instructions on brain alignment. The paper presents a comprehensive analysis of how different instructions from MLLMs capture visual concepts and their correlation with brain activity, using fMRI data from the Natural Scenes Dataset (NSD).", "questions": "None", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper introduces a novel approach to evaluating the brain alignment of instruction-tuned MLLMs, providing valuable insights into how these models process multimodal information in relation to human brain activity.\n- The findings have implications for understanding how brain activity corresponds to the processing of multimodal information, which could be valuable for cognitive neuroscience and AI research.", "weaknesses": "- The study relies on the NSD dataset, where subjects passively view images, which may not fully capture brain activity aligned with task-specific instructions. Active task engagement during fMRI scans could provide a more comprehensive evaluation.\n- How do the authors address ethical considerations regarding the use of fMRI data, especially in relation to participant privacy and data security?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LBWxU63VLr", "reviewer": "Reviewer_6bnp", "summary": "The authors test the ability of instruction tuned multimodal LLMs to match human visual cortex responses to static scenes in the NSD. They compare instruction tuned models to one standard multimodal model (CLIP) and a unimodal vision model.\n\nThe findings reveal that all multimodal models match visual cortex responses better in both low- and high-level visual regions. There does not seem to be a difference between CLIP and instruction-tuned models in any region. Within the instruction tuned models, there seem to be differences between tasks that best explain different parts of visual cortex, with lower-level tasks (e.g., color) better describing retinotopic areas and higher-level tasks (e.g., image captioning) better explaining higher-level tasks. Variance partitioning reveals that there is shared variance between many pairs of tasks, but some tasks like food labeling and scene recognition are more unique.\n\nIt is difficult to draw strong comparisons about these models compared to unimodal vision models though, because of the many differences (architecture, training data, task). As others have pointed out in prior work, advantages of multimodal models in visual cortex significantly decrease when you consider more balanced pairs (e.g., SLIP family models in Wang et al, and Conwell et al) and it seems likely that many of these differences would also diminish in more controlled modeled comparisons. If there are matched vision transformers trained on the same dataset, including them would strengthen the claims of the paper. \n\nThe task comparisons are interesting, but it would help for the authors to spell out what is at stake in these investigations. Does better fit of one type of instruction mean tell us what tasks to train neural networks on for improving human-aligned AI, or does it tell us something about the tuning properties of visual cortex. It would help to clarify this, particularly in the paper introduction and discussion.", "questions": "-\tIn Figure 2 it seems like CLIP is far outperforming the randomly initialized network. Why is there no asterisks?\n-\tIs Figure 6 summarizing the results across NSD-General? Why show only for one representative subject rather than the average of all?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "-\tInstruction-tuned multimodal models are an interesting way to investigate task tuning, as they have higher performance than prior fine-tuned models like the taskonomy set\n-\tThe comparison of retinotopic versus category-selective tuning is interesting, and may yield novel insights into high-level vision.\n-\tVariance partitioning allows a more fine-grained look into the contribution of different task tuning", "weaknesses": "-\tWhile the instruction model-brain comparisons are very interesting, it is not entirely clear what is at stake. Is the central question spelled out in the intro (lines 83-85) important for better human-alignment of AI, or in order to reveal tuning properites of the human brain? If the latter, the authors should flesh this out, and state some limits of this model comparison approach (see below). The primary goal of the study should be clarified in the introduction.\n-\tThe authors make a distinction between the advantage of multimodal models in high- versus low-level visual cortex. The results look very qualitatively similar in early versus late ROIs in Figures 2 (and also compared to other category selective regions in Figure 10) so if this is a major point, it should be backed up with a statistical analysis, (e.g., non-parametric anova, or permutation-based comparison of the vision-multimodal difference in both regions).\n-\tPrior work has shown that when architecture and training set are matched (e.g., the SLIP family models) advantages of multimodal models largely go away (e.g., Wang et al 2023, Conwell et al., 2022). It seems likely that the advantage of the multimodal models over unimodal vision models in this paper is similarly due to different/richer training data, rather than multimodality itself.\n-\tThe distinction between the different instructions/tasks could be made clearer. It seems like the tasks vary from low- high-level and many of the papers claims suggest this distinction. It would help to explicitly order the instructions and make this distinction clear early in the paper (e.g., Table 1). Without a priori labels for low- versus high-level tasks, some of the conclusions seems somewhat circular (eg a task that best explains retinotopic cortex is low-level). This ordering could also help make the results in Figure 3 more clear.\n-\tIt is difficult to see trends that are constant across the two different instruction-tuned models in Figure 3 and 11. InstructBLIP seems to show the trends highlighted in the paper, mPLUG-Owl looks quite random. Perhaps the re-ordering of the tasks/color bar suggested above will help. Alternatively, perhaps small differences across the models are being magnified in the winner-take-all plot. Either way, the authors should address these discrepancies.\n-\tAs a small point, the ROI labels in Figure 2 could be more intuitive. I believe “whole brain” refers to the NSDGeneral mask? If so, this should clarify that it is only visual cortex, not whole brain. pRF-Visual and FLOC-PLACES would be more clear as “retinotopic early visual cortex” and “scene-selective” or something similar\n-\tIt is unclear what Figure 5 is adding to the paper. Layerwise comparisons in different transformer networks are somewhat confusing and also seem dependent on the structure of encoders/decoders. The authors should consider moving this to the supplement. If it is important to the main findings, the author should clarify how the layerwise comparisons relate to the main text, and how they should be interpreted in light of any architectural differences across models (eg encoder vs decoder layers)\n-\tThe variance partitioning is a major strength of the paper, but it seems to only investigate shared variance between pairs of tasks. The more important question seems to be how much (if any) unique variance is explained by models tuned on any one task? And what these results can tell us about the brain.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "X5InECABZS", "reviewer": "Reviewer_vZ7u", "summary": "The paper studies the correlation between multimodal LLMs (MLLMs) and fMRI from the Natural Scenes Dataset (NSD). Specifically, they feed images and instructions to the MLLMs, cache the embeddings, and fit a linear model to map from MLLM embeddings to fMRI. They find that MLLMs exhibit higher brain alignment than vision-only models and CLIP. They also analyze the correlation of specific instructions and specific MLLM layers with brain regions. Finally, they do a variance partitioning analysis to quantify the overlap between different pairs of instructions.", "questions": "- How does the CLIP baseline work?\n    - L235 states “we input both image and ground truth caption pairs” — are the CLIP image and CLIP text embeddings simply concatenated? Are you using the final pooled output from CLIP, or some intermediate layer? If you are using an intermediate layer, how do you pool across image patches / tokens?\n- In L29, what does “effectively capture” mean?\n    - Does this mean that the MLLM embedding correlates with the “expected” brain region that is known to do a certain type of processing? This is not obvious from the discussion in L365.\n- Below are a few minor comments.\n    - L280: typo; “random **initialization** of the 3 MLLMs”\n- Below is a comment on the limitations section, although this note is not important for my rating and I leave this to the discretion of the authors.\n    - I wish the paper would also briefly discuss the limitations of fMRI itself, and how it is not exactly synonymous with “processing in the brain.” Specifically, fMRI is imprecise, as it “measures a surrogate signal” where the “spatial specificity and temporal response” is constrained [1].\n    - Nevertheless, fMRI is the most available / accessible brain signal to study, and it is still interesting to study the relation between machine learning models and fMRI.\n\nReferences\n\n[1] Logothetis, N. What we can do and what we cannot do with fMRI. *Nature* 453, 869–878 (2008).", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The central question the paper studies (the alignment of MLLMs and fMRI) is novel, as MLLMs have not been studied in prior work. The paper is also interesting because it presents some insights into the internals of MLLMs, for example, the fact that image captioning overlaps highly with other instructions.\n- The presentation of the paper is good. The experiments are well-motivated and interpreted with precise language. The paper is also thorough in its setup and appropriately references prior work.\n    - The ablations relating specific instructions or model layers to brain regions (Sec. 6.2) were interesting, even if for some instructions the association was inconclusive.\n    - Ablations, such as the baseline “cross-subject prediction accuracy” (L174) and “variance partitioning approach” (L262), are motivated by prior work. The usage of a ridge regression based model (L242) and Pearson Correlation as a metric (L252) for brain alignment is also standard practice.", "weaknesses": "- The experiments could include additional ablations for the input to the MLLM.\n    - For example, similar to the setup in Sec. 6.1, one could ablate feeding only the image or only the instruction to the MLLM, which reduce to “vision-only” or “LLM-only” baselines. This could help control for model size / other model statistics, i.e., ViT-H might also perform worse because it is a smaller model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)