Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction
Authors: Guobin Shen
Dongcheng Zhao
Xiang He
Linghao Feng
Yiting Dong
Jihang Wang
Qian Zhang
Yi Zeng
Venue: NeurIPS 2024 poster (2024)
Keywords: Neural decoding
Mind Reader
Visual Reconstruction
Multimodal Large Model
Concept Localization
Primary area: neuroscience_and_cognitive_science

Abstract: Decoding non-invasive brain recordings is pivotal for advancing our understanding of human cognition but faces challenges due to individual differences and complex neural signal representations. Traditional methods often require customized models and extensive trials, lacking interpretability in visual reconstruction tasks. Our framework integrates 3D brain structures with visual semantics using a *Vision Transformer 3D*. This unified feature extractor efficiently aligns fMRI features with multiple levels of visual embeddings, eliminating the need for subject-specific models and allowing extraction from single-trial data. The extractor consolidates multi-level visual features into one network, simplifying integration with Large Language Models (LLMs). Additionally, we have enhanced the fMRI dataset with diverse fMRI-image-related textual data to support multimodal large model development. Integrating with LLMs enhances decoding capabilities, enabling tasks such as brain captioning, complex reasoning, concept localization, and visual reconstruction. Our approach demonstrates superior performance across these tasks, precisely identifying language-based concepts within brain signals, enhancing interpretability, and providing deeper insights into neural processes. These advances significantly broaden the applicability of non-invasive brain decoding in neuroscience and human-computer interaction, setting the stage for advanced brain-computer interfaces and cognitive models.

Neuro-Vision to Language: Enhancing Brain Recording-based Visual Reconstruction and Language Interaction
Guobin Shen 1,2,3,4 Dongcheng 2.23 Xiang He 1,355 Linghao Feng 1.3
Yiting Dong 122,,44 Jihang Wang 1.5 Qian Zhang 1,2,3,5 and Yi Zeng' 1,2,3,4,5t
Brain-inspired Cognitive Intelligence Lab, Institute of Automation, Chinese Academy of Sciences, 2 Center for Long-term Artificial Intelligence,
3 Beijing Key Laboratory Artificial Intelligence Safety and Superalignment, School of Future Technology, University Chinese Academy of Sciences,
5 School of Artificial Intelligence, University of Chinese Academy of Sciences
shhenguobin2021, zhacdongcheng22116, hexiang2021, fengl Sooglingghn inggho2222,, dongyiting2020, wangjihang20211, q. zhang, yi.zeng| @ @ia. .c.cn
Abstract
Decoding non-ine non-invasive brain recordings is pivotal for advancing our understanding of human cognition but faces challenges due to individual differences and complex neural signal representations. Traditional methods often require customized models and extensive trials, lacking interpretability in visual reconstruction tasks. Our framework integrates 3D brain structures with visual semantics using a Vision Transformer 3D. This unified feature extractor efficiently aligns fMRI features with multiple levels of visual embeddings, eliminating the need for subject-specific models and allowing extraction from single-trial data. The extractor consolidates multi-level visual features into one network, simplifying integration with Large Language Models (LL.Ms). Additionally, with diverse AMMR-image-rclated textual data to support multimodal large model development. Integrating with LLMs enhances decoding capabilities, enabling tasks such as brain captioning, complex reasoning, concept localization, and visual reconstruction. Our approach demonstrates superior performance across these tasks, precisely identifying language- based concepts within brain signals, enhanc- ing interpretability, and providing deeper insights into neural processes. These advances significantly broaden the applicability of non-invasive brain decoding neuroscience and camantcomppter interaction, setting the stage for advanced brain-computer interfaces and cognitive models.
have enhanced the fMRI dataset
Introduction
The decoding non-invasive brain recordings, such as those obtained from IMRI (0, a corner- stone of cognitive neuroscience 2 3 This process offers unparalleled insights into the neural underpinnings human cognition, contributing not only to fundamental scientific knowledge but also to advancements in clinical and technological applications. Despite its potential, the field faces significant challenges primarily due to the high variability of brain activity across individuals (5) and the complexity inherent in the neural representations of cognitive processes [6].
Brain decoding techniques have traditionally relied on customized, subject-specific models 7 6
These models necessitate intricate and costly experimental setups, depending on multiple trials
CCorresonndigg Author.
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

to achieve reliable results. Such approaches, while helpful, are inherently limited in scalability and flexibility, hindering broader application and generalization across different populations and conditions.
Visual reconstruction aims recreate perceived visual stimuli from brain signals and is consid- ered one of the benchmarks of brain decoding. However, this approach often struggles to accurately reproduce the visual experiences of individuals, generally lacking semantic precision and interpretabil- ity [8. This inability to effectively decode and reconstruct signals restricts our understanding of how sensory information is processed. Recognizing these limitations, our study introduces language modalities as a critical enhancement designed to assess decoding performance more effectively and enrich brain-computer interfaces' interaction capabilities.
Addressing these multifaceted challenges, our research introduces the Vision Transformer 3D (ViT3D) (10) specifically tailored the domain of visual reconstruction. Unlike traditional ap- proaches that often reduce complex brain regions to one-dimensional vectors [T] 6 12] 13], losing critical spatial structure information, our implementation of VTT3D preserves the dimensiamnni structural integrity of the brain data. This adaptation enables an unprecedented enhancement in the extraction of visual semantic information, ensuring a deeper fidelity and richness in the decoded visual representations.
Our fMRI feature extractor includes unified network backbone and two alignment heads for feature matching. This setup enables efficient, high-quality visual reconstructions across subjects from one experimental trial. By simply aligning the extractor's output with CLIP embeddings [14] and features of Variational Autoencoder (VAE) [15) our method eliminates the need for multiple, subject- specific models, substantially simplifying the decoding process. This straightforward and effective configuration reduces the resources required for brain decoding and showcases the potential for easy integration with Large Language Models (LLMs), enhancing its usability across various applications.
Moreover, our research delves into the integration of brain recordings with visual and linguistic data within a comprehensive multimodal framework using LLMs. This integration significantly improves visual reconstruction performance and introduces the groundbreaking capability for direct interaction through natural language. Our model facilitates diverse communication with brain data using natural language and precisely localizes linguistic concepts within brain signals. These advancements help deepen our understanding of the interactions between language, perception, and neural activity. Additionally, to bolster the development of these multimodal models, we have augmented the brain- recording visual dataset with natural language enhancements.
In summary, our contributions can be summarized as follows:
Our fMRI feature extractor, based Vision Transformer 3D, aligns fMRI features with visual embeddings at multiple levels, integrating 3D brain structures with visual semantics. This eliminates the need for subject-specific models and enables efficient data extraction from single trials, significantly reducing training costs and enhancing practical usability in real-world scenarios.
We expanded the language dimension of our fMRI-visual dataset to build a multimodal large model capable of decoding fMRI data. This enhancement boosts brain decoding performance and broadens the application scope to include tasks like visual reconstruction, question-answering and complex reasoning while also allowing precise localization and manipulation of singuage-based concepts within brain signals.
Experimental results on the Natural Scenes Dataset (NSD) [16] for visual reconstruction and language interaction tasks demonstrate that our method surpasses existing models, effectively achieving concept localization and elimination.
2 Related Works
Non-invasive techniques such as functional magnetic resonance imaging (fMRI) are pivotal in pro- viding direct insights into neural activities, significantly deepening our understanding of complex cognitive processes from neural network structures [17] to advanced image and language processing This section reviews key developments in fMRI-based brain decoding, particu-
tasks [88 19 larly emphasizing the shift from simple subject-specific analyses to more complex, multimodal interpretations.

---

Visual Reconstruction from Brain Activity Visual reconstruction from brain activity involves translating brain recordings into the visual stimuli perceived by subjects. Early methods, like those developed by Horikawa el al. [88] relied on sparse linear regression to predict features extracted by convolutional neural networks from fMRI data. Recent advancements in generative artificial intelligence, particularly diffusion models (20), have propelled efforts to reconstruct visual stimuli directly from fMRL. For instance, Lin et al. [21] aligned fMRI data with image features and corresponding CLIP embeddings to facilitate image generation using fine-tuned StyleGAN (22). Similarly, Takagi et al. [9] improved the quality of visual reconstructions by aligning fMRI with CLIP text embeddings and the latent spaces of diffusion models. Xia et al. [11] aligned fMRI data from dimensions of image CLIP features, depth, and color using T2I Adapters [23] for fine- ane-ggained conditional control. Despite these advancements, the complexity of such methods, involving multiple independent modules, complicates their integration with technologies like LLMS and restricts their generalizability across different subjects. We observed that some contemporary works also attempt cross-subject alignment; however, these methods either require subject-specific parameters [24] or face performance issues compared to subject-specific models 25
IMRI Data Processing Efficiently processing IMRI data to extract visually relevant activities typically involves simplifying the data into one-dimensional vectors and selecting voxels most responsive to visual stimuli. Traditional methods utilize simple linear regression or fully connected networks to predict visual stimulus features [1821] However, these methods often lose essential spatial structural information, which is critical given the individual differences in brain anatomy. To address these challenges, innovations such as Vision Transformer 3D (ViT3D) have been developed for managing data with intricate spatial structures [26] 27]. ViT3D segments 3D data into patches, preserving local spatial information within each patch and maintaining overall structural integrity through self-attention mechanism (28). thereby enhancing the performance of brain activity extraction (2)
Multimodal Integration with Brain Signals The utilization of language as a medium for repre- sentation allows for the expression of complex concepts with precision and abstraction. The advent of LLMS has showcased their potential to act as bridges across different modalities, enhancing interactions with visual and audio data through natural language 31]. For example, approaches like those by Défossez et al. (32). which align brain recordings with spoken language to decode speech non- innaisivvvll, have demonstrated the effectiveness of combining brain recordings with LLMs. However, these approaches are often limited by the specificity of the fMRI feature extractors used, which can restrict the scalability and the size of the models employed. By integrating our specially designed cross-subject fMRI feature extractor, we enhance visual reconstruction quality and enable complex reasoning and direct interaction with model outputs using natural language, achieving precise localization of open natural language concepts within the brain.
3 Methodology
Our approach is designed to tackle the key challenges encountered in the visual reconstruction of brain activity and the integration of LLMs with multimodal data. Traditional brain decoding methods, especially those involving fMRI, often struggle with the complexity of accurately reconstructing visual stimuli and generalizing these models across different subjects. Furthermore, while L.LMs hold significant potential for enhancing interactions across various cognitive models, their integration with neuroimaging data has been hindered by the need for non-scalable or efficient customized, subject-specific models.
In the following sections, we detail our methodology's components, as seen in Fig. I We describe the architecture of our feature preprocessor that maintains the spatial structure of IMRI data, our unified fMRI feature extractor, and the integration strategies for the network with LLMs. We also elaborate on the multimodal interaction techniques that enable direct and meaningful communication between the computational model and the neural representations and the implementation details for visual reconstruction.
3.1 fMRI Feature Preprocessor
fMRI quantifies changes the boood-orygen--eell--eeeeeeddd (BOLD) signals to characterize neural activity. The BOLD signal for given subject can be represented as 3D matrix burigin E Xx,,X,,2, where s indexes the subject, accounting for inter-individual variability. Traditional preprocessing methods typically involve masking voxels sensitive to the specific task, followed by flattening the
3

---

(b)
(c)
Answers 4j
Reconstruct
Transformer Layer PN
We
We
Large Language Model
eaciigg grass
Transformer #f1, N-11
E
3D AMRI Preprocessor
ft
Tokenizer Embeddings
Fooilooning the comccgg
A.l locades the reglons
Instructions 41
Subject
AM
Feature Alignment
FMRI Feature Extractor
Multimodal Interaction via LLMs
MLP Projector
Stable
[CLS) Token
Application
Image Encoder
Diffusian
fMRI Taken
Examples
Overview of the integrated multimodal framework combining IMRI feature extraction with LLMs for interactive communication and reconstruction. The architecture comprises: (a) dual- stream pathway for feature alignment with VAE and CLIP embeddings. (b) A 3D fMRI preprocessor and an fMRI feature extractor. (c) A multimodal LLM integrated with fMRI. The extracted features are then fed into an LLM for processing natural language instructions and generating responses or visual reconstructions.
Figure I:
remaining data into a ID vector. For subject the processed fMRI signal can thus be represented as b, E RIXN, where N, denotes the number of voxels selected after masking.
However, this approach results a loss of spatial structural information, complicating alignment across differ- subjects. We propose feature ex- traction method that preserves spatial structure, shown in Fig. Starting with the original BOLD signal borigin- we first use trilinear interpolation to resize the data to uniform dimension, ensuring maximal spatial consistency across subjects' brains while not in- troducing subject-specific parameters. After resizing, the normalized data un- dergoes patching process where it divided into smaller cubic segments, each dimension y"3 This step retains the local spatial features within each segment, preserving the 3D structure crucial for accurate analysis. Finally, patches containing non-task-relevant information are filtered out to reduce computational load. This results in patched data with dimensions IR NCC, where N is the number of patches retained that contain meaningful information. The entire preprocessing operation can be summarized mapping:
trilncar inteppsttion
panching
Subject
Active patcls
Inactive paachs
Figure 2: Description of fMRI data preprocessing. First align
the data of different subjects, then patch them, and finally remove activity-irreleaan patches.
{p= Variinn + Vorigi E X,XYXXX, E
In Eq. [ p is the preprocessing function applies resizing, patching, and masking to transform the original fMRI data into structured format. This function provides uniform representation of the BOLD signals across different subjects, ensuring that the spatial structure is preserved and capable of integration with Transformer architectures.
3.2 Dual-Stream fMRI Feature Extractor
Visual reconstruction tasks typically begin by mapping the processed BOLD signal, b, to various estimated visual feature spaces, represented as (ε), .., where indicates different levels of visual features and é denotes the visual features estimated from the BOLD signals. Subsequently, these features are used to reconstruct the image, represented as é, from the visual features. To enhance the quality of the reconstructed images, complex feature extractor designs are required,

---

which increases the preprocessing and computational overhead. However, thanks to the design of our fMRI feature extractor, we can achieve efficient visual reconstruction using single network backbone, as shown in Fig. Tb).
Specifically, the patched features obtained from Eq. 1 are directly processed through Transformer Encoder E, to extract features, obtaining the hidden states from the last layer, Λ = (,),), where No represents the number of layers of the encoder. These outputs are then aligned with the visual
stimulus's CLIP embeddings že (x) and VAE features žn (z), where I represents the visual stimulus. The loss function used to train the fMRI feature extractor can expressed as:
Calign ((b.)) [|55uc (10%)
+ -
[w -
(2)
In Eq. the expectation (b, x) X) averages the alignment loss across samples from the B (fMRI signals) and X (corresponding visual stimuli). Here, h represents the output from the first token of the last hidden state layer of the encoder &g. The functions fw, and fa. are two-layer perceptrons designed to align the extracted fMRI features with these embeddings. The hyperparameter balances the losses between the alignments of CLIP and VAE features. Through this dual-stream configuration, we create a backbone network that incorporates different levels of visual features. Using only simple MSE loss function, achieve high- high-quality visual reconstruction.
3.3 Multimodal Interaction with fMRI
Our feature extractor architecture, equipped with single backbone network, adept at encapsulating various feature levels, making it highly suitable for integration with LLMs. Inspired by advancements such as those in LLaVA [33], we utilize the penultimate hidden states of our network, h A,N,-1 as multimodal tokens of fMRT data. A two-layer perceptron fa projects this state to the same dimension as the text embeddings, resulting in the fMRI embeddings i (hoNo1 Considering a sequence of question-answe pairs related to the fMRI data (4o, 4o Q1, 01...- -.. 9L, at), the training objective is formulated as:
max Po (A (A),()) max II (a (aa, 11-1.-.- 2o. t)- ==0
Eq. 3 describes the probability of generating sequence of answers A given a sequence of questions Q and the derived fMRI embeddings t. Each answer is conditionally dependent on all preceding questions and answers, well as the context embeddings derived from fMRI data, where @ represents the trainable parameters of LLMs.
To effectively couple fMRI data with language models, annotated data is essential. Although the NSD [16] uses labeled visual stimuli from the COCO dataset [34), semantic mismatches often occur due to modifications such as image cropping performed when displaying images to subjects AAppendixAAII.. Morcover, the original captions in NSD are not detailed enough to capture nuanced semantic information. Recognizing the importance of comprehensive linguistic annotations, have constructed diverse instructional dataset that includes various textual data: brief descriptions, detailed descriptions, continuous dialogues, complex reasoning tasks, instruction reconstruction, and concept localization.
3.4 Interaction and Reconstruction via LLMs
Our fine-tuned model can understand information embedded within fMRI data and adhere to human instructions. Interactions and explanations of visual stimuli content occur through natural language. A typical dialogue format is structured as follows: <human> : : [image] [instruct ruction] <bot>: [answer]. Here, [instruction inference is tokenized and embedded, while [image] ] acts placeholder, replaced by the fMRI data embedding t. The model then responds based on the directive and the embedded fMRI data. [answer] represents the response generated by the LLMs.
denotes natural language instruction, which during
After instruction-basee fine-tuning, the model communicates directly via natural language and supports visual reconstruction and location identification of concepts expressed in natural language, as shown Fig. 3 These are facilitated respectively through Stable UnCLIP (0) for visual reconstruction and GradCAM [35) for concept localization.
5

---

Concept Locking
fiele?
nany
cheagoinn is 1ikely doing with the rowte control based on their postare and the surrouncing enu rosset?
cammopenting Stuble proupte for the laoge.
di Locating "gtratte"
There stout twenty Grazteg.
C: can you descrise the positioning
The person is likely witching corteolling
orign, calania tai., modes Jun
Lorating the concept....
Referance
Fer
For Reference
spread closer the trees while others ane further away.
the ruecto control. they ane holdiing ie and the cory setting with blanket suggests relasse and
Figure 3: Demonstration of the model's 's capabilities for engaging in multi-round dialogue, complex reasoning, visual reconstruction, and concept location tasks using fMRI data.
For visual reconstruction, the LLM initially generates a reconstruction prompt ay, which, combined with the latent representations 2n and & from the fMRI feature extractor, results in the generation of an image. This process can be formalized as:
D B) Bo ac, a E (V,,
In Eq. 4. D represents the frozen UnCLIP, used for visual reconstruction, where & and a, serve as conditional information during the denoising process, and é, acts as the initial latent representation of the image. The hyperparameter / is used to introduce noise into the latent space prior, balancing low- level features brought by the prior and high-level information controlled by the diffusion conditions during the denoising process. For concept localization in natural language, LLMs activate the feature extractor using keywords from the instructions, enabling precise location identification of the discussed concepts.
4 Experiments
4.1 Implementation Details
Dataset and Preprocessing: We utilized the Natural Scenes Dataset (NSD) [6]. containing high- resolution 7Tesla fMRI scans and corresponding visual stimuli from COCO 34 The dataset involved eight subjects, but analyses focused on the four (subj01, subj02, subj05 and j07) who completed all sessions. Modifications like cropping necessitated re-annotation of images using BLIP2 (36] for captions and DETR (37] for bounding boxes to maintain consistency. fMRI data was standardized dimensions 83 104 81 using trilinear interpolation and segmented into 14 x 14 X 14 patches.
Architecture and Training: Our architecture integrates CLIP ViT-L/14 14] and an Autoen- coderKL [15] for image feature extraction, aligned with fMRI data processed through a 16-layer Transformer Encoder (28). This setup employed two perceptrons (fw.. fw..) to align features with CLIP and VAE, respectively. Training involved a multi-stage approach where the LLM was initially frozen, followed by fine-tuning stage for both the LLM and the Transformer Encoder. For visual reconstructions, the model utilized UnCLIPP22 20 with 3 set to 0.93, and concept localization was achieved using GradCAM (35]. For more details the dataset and experimental setup, please refer to Appendices A and C Additional experimental results can be found in Appendix D
4.2 Captioning and Question Answering
Tab. shows the performance of our method multimodal language tasks. With the introduction of L.Mss, we have expanded the task forms to include brain captions, detailed descriptions, and complex reasoning, as illustrated in Fig. 3 Our approach has demonstrated superior performance on the majority of metrics for the brain captioning task. Notably, our model can generalize across subjects without the need to train separate models for each subject or introduce subject-specifie parameters.
Given the semantic mismatch between captions and images in the original NSD /Section3.3) we reran the experiment using BLIP2 [36] generated captions ground truth. The results, shown in Tab. I show significant improvements when evaluated against BLIP2-generane captions, confirming the effectiveness of our model in the brain captioning task and the reasonableness of the task setting.
Beyond brain captioning, we have incorporated tasks for detailed description and complex reasoning. Our model also achieved the best performance these two tasks, suggesting that it can generate

---

Table 1: Quantitative analysis of brain captioning, detailed descriptions, and complex reasoning tasks. Some results are derived from UMBRAE [4] Results are compared to those from other studies, with best
second and third highlighted. Underline indicates the best result under identical conditions, while denotes results obtained using #LIP2-generated captions as ground truth.
Method
Models
BLEU1
BLEU2
BLEU3
BLEU4 METEOR
ROUGE
CIDEr
SPICE
CLIP-S
Brain Caption
SDRecon
36.21 47.04
17.11
7.72
3.43
10.03
25.13
13.83
5.02
61.07
Onel.IM 18
26.97
13.49
9.51
13.55
35.05
22.99
6.26
54.80
UniBrain
16.90
BrainCap 40
G6.96
36.21
22.70
14.51
16.68
40.09
41.30
9.06
64.31
UMBRAE
1
57.84 84
38 43
23.41
17.17
18.70
42.14
53.87
12:27
66. 10
Our Method
57.19
37.17
23.78
15.85
18.60
36.67
49.51
12.
65,49
w/o VIT3D
52.91
15.64
8.49
14.07
39.64
56.92
Our Method
64.26
51.44
47.70
32.17
20.41
52.67
83.94
18.27
68. 78.72
w/o VIT3"
58.87
42.11
29.48
21.39
15.85
38.48
56.37
11.27
64.35
Detail Description
Our Method
38 91
24.02
15.24
12.41
18.44
27.83
42.58
18. 41
56.
w/o VTT3D
1
33.57
18.95
11.09
6.13
15.56
23.80
20.23
16.21
51.47
Complex Reasoning
Our Method
65.41
59.61
50.68
36.46
34.46
62.60
217.83
60.29
80,96
w/o VIT3D
1
60.36
47.81
39.76
30.57
21.37
45.30
150.67
52.13
73.26
not only simple captions but also detailed descriptions and perform complex reasoning. The model's performance increases complex reasoning tasks, possibly due to the richer semantic information in the questions, which our model captures more effectively. An ablation study was also conducted, revealing noticeable performance drop in multimodal language tasks when the cruuctural-prseervii features of fMRI data were not extracted using VIT3D. Instead, the fMRI data were flattened and patched, similar methods used in other literature, while maintaining the same fMRI feature extractor structure. This underlines the effectiveness of ViT3D and the capability of our model in multimodal tasks.
4.3 Visual Reconstruction
While our primary objective extends beyond mere visual decoding from fMRI data, visual recon- struction offers tangible demonstration of a model's comprehension of fMRI data. Therefore, we conducted visual reconstruction experiments and compared our results with those from other studies. The quantitative evaluation highlights our method's proficiency.
Tab. [ showcases that our model competes with or surpasses traditional subject-specific frameworks on several metrics. Notably, it excels in high-level feature matching, demonstrating the model's ability to effectively leverage LLMs for interpreting complex visual data. The robust performance across various visual stimuli confirms our model's comprehensive understanding of fMRI data, Experiments without key components like LLM and VAE features highlight the significance of each element in our method, which is crucial for achieving state-of-the-a results. Moreover, we have conducted single-trial experiments, opting to use only the first visual stimulus, similar to the approach of MindEye (6), rather than averaging signals from three identical stimuli, which typically escalates data collection costs. Even under these more stringent conditions, our model shows only slight decrease in performance, enhancing its feasibility for practical applications. Visual reconstruction examples are provided Fig. 4 illustrating the effectiveness of our approach.
The balance between noise introduction and feature preservation in IMRI data visual reconstruction is governed by the hyperparameter B. F ppresents detailed ablation study on how different 3 values impact various metrics. Fig. ((a)) shows that Pixel Correlation (PixCorr) peaks at intermediate B values, indicating the optimal balance between injected noise and retained prior. The integration of the
LLMs does not significantly influence low-level feature capture. In Fig. ((bb). increasing & enhances CLIP accuracy, with LLM integration having substantial effect on capturing high-level features. Fig. c)) indicates the features of the 5 th layer of AlexNet, similar to CLIP features, effectively represent the similarity between reconstructed images and visual stimuli, capturing high-level features accurately. Additionally, Fig. d) illustrates that both the Structural Similarity Index (SSIM) and
7

---

Table 2: Quantitative evaluation visual reconstruction. Performance metrics are reported across different levels of features, with the best second and third scores highlighted. The underline indicates best result under the same conditions. Our method achieves state-of-the-art results using single model trained across subjects (# Models 1).
Method
# Models PixCor SSIM AlexNeu(2) 1 AlexNet(S) Inception
Low-Level
High-Level
CLIP EffNet-B SWAV J
Mind-Reader 21
78.2%
Takagi et al 9
83.0%
83.0%
76.0%
77.0%
Gu er 41
.150
.325
.862
.465
42
.254
.356
94.2%
96.2%
87.2%
91.5%
.775
.423
MindEye
4
.309
.323
94.7%
97.8%
93.8%
94. 1%
.645
.367
DREAM 111
4
.288
.338
95.0%
97.5%
94.
95.2%
.638
.433
MindBridge 25
.151
.263
87.7%
95.5%
92.4%
94.7%%
.712
.418
UMBRAE
1
.283
.328
93.9%
96.7%
93.44%
94.1%
.700
.393
Method
.265
.357
93.1%
97.1%
96.
07.5%
321
w/o L.LM
.263
.369
92.0%
17.1%
94.2%
96.1%
.680
.328
w/o VAE feature
.093
.263
84.5%
90.6%
93.6%
95.7%
.684
.398
w/o C_Subj
.241
.356
88. 1%
95.7%
92%
95.1%
.631
.347
w/o C_Subj & VIT3D
.164
.273
86.7%
91.4%
89.3%
91.8%
.731
.417
Single Trial
MindEye 6 Our Method
.255
.308
91 01.6%
95.9%
91.3% 94.6%
91.6% 95.3%
.601
.398
.257
.336
91.2%
96.3%
.671
.324
Visil
Mird 2023
Brr biftuer 60031
Tolagi et
bepwe 13t04
Voal
Winddye (2228)
Mathod
Misa Simao
Mecte
Ou Methol
(a)
(b) Single
Figure 4: Visual reconstruction results showcasing the comparison between (a) using the average signal from all trials and (b) using the first visual stimulus.
CLIP scores benefit from optimally chosen / values, with LLM integration enhancing overall image quality and semantic accuracy. Appropriately adjusting 3 helps balance the representation of different feature levels in the reconstructed images. Fig. 6 provides examples of visual reconstructions various 3 values, demonstrating the model's enhanced capabilities.
4.4 Concept Localization
To further our understanding of semantic concept localization within brain signals, capitalized on the alignment between our IMRI encoder and CLIP features, which were developed during the training phase. Building on this, we devised a method localize concepts within brain signals. Specifically, we first fine-tuned Language Models (LL.Ms) to extract the target concepts from natural language. These concepts, once encoded through the CLIP text encoder, served as targets for GradCAM, which facilitated the localization of the concept within brain signals. To enhance the precision of our localization, we trained three models with varying patch sizes (14, 12, 10) and utilized the penultimate layers of all models to extract semantic features. Fig. 7 illlssraates the brain signal localization results for different semantic information. indicating our method's capacity to discriminate the positions of various semantics within brain signals for the same visual stimulus.

---

0.6
0.-
11.02
0.8
0.0
01000
0.7
0.7
0.82
0.6
0.6
0.7
0.8
alesmett)
cip
Figure Ablation analysis of the hyperparameter visual reconstruction performance.
0.5
0.75
0.98
=0.6
0.75
0.9
0:98
1.0
Visual Stimulus
Visual Stimalus
+
LUMe
Figure 6: Visualization of the impact of B on visual reconstruction.
To validate the efficacy of our method, conducted an ablation study the semantic concepts. After localizing the concepts in the original brain signals, we zeroed out the signals in the identified voxels and performed feature extraction and visual reconstruction using the modified brain signals. As depicted in Fig. g the removal neural activity in specific brain regions associated with certain semantic concepts resulted in the omission of the corresponding semantics in the visual reconstruction. This substantiates the validity of our concept localization method within brain signals and demonstrates our approach's 's capacity for extracting and modifying semantic information in brain activity, which is pivotal for comprehending semantic information processing in the brain.
Conclusion
Our study has successfully developed and validated novel brain decoding framework that leverages the capabilities of Vision Transformer 3D in conjunction with IMRI data, enhanced by the integration of LLMs. This approach has demonstrated notable improvement in the reconstruction of visual stimuli from brain signals, offering more precise and interpretable understanding of the underlying neural mechanisms. The experimental results confirmed the robustness of our model in performing various cognitive tasks, including captioning, cvestion-answerrie and visual reconstruction, all from single-trie triil fMRI data. By enabling accurate localization of linguistic concepts within the brain, our work has potential applications in developing brain-computer interfaces and advanced cognitive modeling. Conclusively, this research contributes to the broader endeavor of decoding and interpreting brain activity, with significant implications for neuroscience and technology interface development. The fusion of advanced AI models with neuroimaging opens new avenues for exploring intricacies human cognition and the seamless integration technology with neural processes.
'cow'
'tree'
'a aarplone
'doue'
motorrycle
'rood'
'cat'
bench
Figure 7: Differential heatmaps of neural activity representing various semantic information for the same visual stimulus.

---

Tecominucticm
Pecondructte
orgiinal
Iegghnl
bandlf
orgiinal
Therr
orgg nol
Lac
Figure 8: Validation of concept localization by semantic signal nullification and its effect on visual reconstruction.
Acknowledgments and Disclosure of Funding
This work was supported in part by the Beijing Major Science and Technology Project (Grant No. 7241 120010000)240055
References
[1] Nikos K Logothetis. What can do and what we cannot do with fmri. Nature, 453(7197):869- 878, 2008.
[2] Yiheng Hu and Qing Yu. Spatiotemporal dynamics of self self-generated imagery reveal a reverse cortical hierarchy from cue-induced imagery. Cell Reports, 42(10), 2023.
[3] Russell A Poldrack. The future of fmri in cognitive neuroscience. Neuroimage, 62(2): 1220, 2012.
[4] Russell A Poldrack. The role of fmri in cognitive neuroscience: where we stand? Current opinion in neurobiology, 18(2):223-227,
2008 2008
[5] Zijiao Chen, Jiaxin Qing, Tiange Xiang, Wan Lin Yue, and Juan Helen Zhou. Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22710-22720, 2023.
[6] Paul Scotti, Atmadeep Banerjee, Jimmie Goode, Stepan Shabalin, Alex Nguyen, Aidan Demp- ster, Nathalie Verlinde, Elad Yundler, David Weisberg, Kenneth Norman, et al. Reconstructing the mind's eye: fmriito-imaae with contrastive learning and diffusion priors. Advances in Neural Information Processing Systems, 36, 2024.
[7] Andrew Luo, Margaret Marie Henderson, Michael J Tarr, and Leila Wehbe. Brainscuba: Fine- grained natural language captions of visual cortex selectivity. In The Twelfth International Conference on Learning Representations, 2023.
[8] Jiaxuan Chen, Yu Qi, Yueming Wang, and Gang Pan. Mindgpt: Interpreting what you see with non-invasive brain recordings. arXiv preprint arXiv:2309. 15729, 2023.
[9] Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patlern Recognition, pages 14453-14463 2023.
[10] Jean Lahoud, Jiale Cao, Fahad Shahbaz Khan, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan, and Ming-Hsuan Yang. 3d vision with transformers: A survey. arXiv preprint a208.:2088.0 04309, 2022.
[11] Weihao Xia, Raoul de Charette, Cengiz Oztireli, and Jing-Hao Xue. Dream: Visual decoding from reversing human visual system. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 8226-8235, 2024.
[12] Jingyuan Sun, Mingxiao Li, Zijiao Chen, Yunhao Zhang, Shaonan Wang, and Marie-Francine Moens. Contrast, attend and diffuse to decode high-resolution images from brain activities. Advances in Neural Information Processing Systems, 36, 2024.
10

---

[13] Andrew Luo, Maggie Henderson, Leila Wehbe, and Michael Tarr. Brain diffusion for visual exploration: Cortical discovery using large scale generative models. Advances in Neural Information Processing Systems, 36, 2024.
[14] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference machine learning, pages 8748-8763. PMLR, 2021.
[15] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.61 14, 2013.
[16] Emily J Allen, Ghislain St-Yves, Yihan Wu, Jesse I. Breedlove, Jacob S Prince, Logan T Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, et al. A massive fimi dataset to bridge cognitive neuroscience and artificial intelligence. Nature neuroscience, 25 2022.
[17] Stephen M Smith, Karla L Miller, Gholamreza Salimi-Khorshide Khorshidi, Matthew Webster, Christian Beckmann, Thomas E Nichols, Joseph D Ramsey, and Mark W Woolrich. Network modelling methods for fmri. Neuroimage, 54(2)875-891,, 2011.
[18] Tomoyasu Horikawa and Yukiyasu Kamitani. Generic decoding of seen and imagined objects using hierarchical visual features. Nature communications, 8(1):15037, 2017.
[19] Jerry Tang, Amanda LeBel, Shailee Jain, and Alexander G Huth. Semantic reconstruction of continuous language from non-invasive brain recordings. Nature Neuroscience, 26(5):858-88) 866, 2023.
[20] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pages
2022.
[21] Sikun Lin, Thomas Sprague, and Ambuj K Singh. Mind reader: Reconstructing complex images from brain activities. Advances in Neural Information Processing Systems, 55:29624-296321 2022.
[22] Tero Karras, Samuli Laine, Miika Aittala, Janne Hellsten, Jaakko Lehtinen, and Timo Aila. Analyzing and improving the image quality of stylegan. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 8110-8119, 2020.
[23] Chong Mou, Xintao Wang, Liangbin Xie, Yanze Wu, Jian Zhang, Zhongang Qi, and Ying Shan. T2i-adapter: Learning adapters to dig out more controllable ability for text-to-image diffusion models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 1296-4304, 2024.
[24] Weihao Xia, Raoul de Charette, Cengiz Öztireli, and Jing-Hao Xue. Umbrae: Unified multi- modal brain decoding. In European Conference on Computer Vision (ECCV), 2024.
[25] Shizun Wang, Songhua Liu, Zhenxiong Tan, and Xinchao Wang. Mindbridge: A cross-subject brain decoding framework. arXiv preprint arXiv:2404.078800 2024.
[26] Ali Hatamizadeh, Yucheng Tang, Vishwesh Nath, Dong Yang, Andriy Myronenko, Bennett Landman, Holger R Roth, and Daguang Xu. Unetr: Transformers for 3d medical image segmentation. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pages 574-584, 2022.
[27] Wang Wenxuan, Chen Chen, Ding Meng, Yu Hong, Zha Sen, and Li Jiangyun. Transbts: Multimodal brain tumor segmentation using transformer. In International Conference on Medical Image Computing and Computer-Assisted Intervention, Springer, pages 109-119, 2021.
[28] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, tukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017.
11

---

[29] Yucheng Tang, Dong Yang, Wenqi Li, Holger R Roth, Bennett Landman, Daguang Xu, Vishwesh Nath, and Ali Hatamizadeh. Self-supervised pre-training of swin transformers for 3d medical image analysis. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 40730-207400 2022.
[30] Jean-Baptiste Alayrac, Jeff Donahue, Pauline Luc, Antoine Miech, lain Barr, Yana Hasson, Karel Lenc, Arthur Mensch, Katherine Millican, Malcolm Reynolds, et al. Flamingo: a visual language model for few-shot learning. Advances in neural information processing systems, 35:23716-23736, 2022.
[31] Haotian Liu, Chunyuan Li, Yuheng Li, and Yong Jae Lee. Improved baselines with visual instruction tuning. arXiv preprint arXiv:2310.03744. 2023.
[32] Alexandre Défossez, Charlotte Caucheteux, Jérémy Rapin, Ori Kabeli, and Jean-Rémi King. Decoding speech perception from non-invasive brain recordings. Nature Machine Intelligence, 5(10):1097-1107, 2023.
[33] Haotian Liu, Chunyuan Li, Qingyang Wu, and Yong Jac Lee. Visual instruction tuning, Advances in neural information processing systems, 36, 2024.
[34] Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCC 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pages 740-755. Springer, 2014.
[35] Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das. Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based localization. In Proceedings of the IEEE international conference computer vision, pages 618-626, 2017.
[36] Jumnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- pre-training with frozen image encoders and large language models. In International conference on machine learning, pages 19730-19742. PMLR, 2023.
[37] Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. In European conference on computer vision, pages 213-229 Springer, 2020.
[38] Jiaming Han, Kaixiong Gong, Yiyuan Zhang, Jiaqi Wang, Kaipeng Zhang, Dahua Lin, Yu Qiao, Peng Gao, and Xiangyu Yue. Onellm: One framework align all modalities with language. arXiv preprint arXiv:23/2.03700 2023.
[39] Weijian Mai and Zhijun Zhang. Unibrain: Unify image reconstruction and captioning all in one diffusion model from human brain activity. arXiv preprint 2308::2200.0742283 2023.
[40] Mattco Ferrante, Furkan Ozcelik, Tommaso Boccato, Rufin VanRullen, and Nicola Toschi. Brain captioning: Decoding human brain activity into images and text. arXiv preprint arXiv:2305.1 15560, 2023.
[41] Zijin Gu, Keith Jamison, Amy Kuceyeski, and Mert R Sabuncu. Decoding natural image stimuli from fmri data with surface-based convolutional network. In Medical Imaging with Deep Learning, 2023.
[42] Furkan Ozcelik and Rufin VanRullen. Natural scene reconstruction from fmri signals using generative latent diffusion. Scientific Reports, 13(1):15666, 2023.
[43] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
[44] Meta. Meta llama
2024.
[45] Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions on image processing, (34) 612, 2004.
12

---

[46] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[47] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Re- thinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818 1-2286,, 2016.
[48] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International conference machine learning, pages
14. PMLR, 2019.
[49] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912-9924, 2020.
[50] Lianmin Zheng. Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang. Zi Lin, Zhuohan Li, Dacheng Li, Eric. P Xing. Hao Zhang, Joseph E. Gonzalez, and lon Stoica. Judging llm-as-a-judge with mt-bench and chatbot arena, 2023.
13

---

A Dataset
A.1 Natural Scenes Dataset
We conducted our experiments on the Natural Scenes Dataset (NSI) 16 which consists of high- resolution 7Tesla fMRI scans collected from eight healthy adult participants. Our analysis focused on the four subjects (subj01, subj02, subj05, and j07) who completed all data collection sessions. Participants during the sessions. However, the NSD required preprocessing to correct temporal resampling for slice timing differences and spatial interpolation to adjust for head motion and spatial distortion. We processed the scans in 1.8-mm native volume space, particularly targeting the "nsdgeneral" region known for its high sensitivity to visual stimuli. This area, predominantly covering the posterior cortex, is crucial for targeted analysis of visual processing.
exposed to thousands of natural scene images from the COCO dataset [34]
There a laptop computer in room with couch and bookcase.
person kite boarding in rough seas near the shoreline.
A jeep with dead bird the windshield.
three people riding horses beach
A bathroom with sink, mirror, and lights in it.
the seven sheep are preparing to be sheered
A toilet, mirror and sink in toilet
A three car passenger train on the train tracks.
Figure 9: Examples of some images and corresponding captions from the NSD dataset. Due some image operations such as cropping, there a mismatch between the original captions and the instance bounding boxes.
Our testing protocols included using the average response from three trials associated with each image to enhance reliability, a common practice in recent studies, as well as assessing each response separately provide more challenging and practically relevant test scenario. This approach allowed us to rigorously evaluate our method under realistic and diverse conditions, ensuring thorough validation against established benchmarks. Modifications such as cropping when presenting images to participants led to mismatches between the original captions and instance bounding boxes, as illustrated in Fig. g To ensure data consistency, we re-annotated the cropped images, generating eight captions for each image using BLIP2 [36].
A.2 Language Extension
To ensure compatibility between IMRI data and Large Language Models (LLMs) and enable instruction-folloiii and diversified interactions, we extended the Natural Scenes Dataset (NSD) with natural language annotations. This extension includes seven types of dialogues: brief descriptions, detailed descriptions, continuous dialogues, complex reasoning tasks, instruction reconstruction, and concept localization.
first generated concise descriptions the visual stimuli using BLIP2 and integrated these with the original COCO dataset captions to create brief descriptions of the images. Subsequently. DETR 337 was employed to generate bounding boxes for these images. We then combined the image captions and bounding box information inputs and utilized GPT-3. 3. tuuboo--oo---- -0125 to generate various
14

---

forms of dialogues. These dialogues were manually adjusted for format and content to ensure consistency and relevancy. For specifies on the prompts used during generation, please refer the supplementary document.
Furthermore, constructed multimodal fine-tuning dataset based the generated dialogues. For each piece of fMRI data, we created corresponding language extensions. Commands for brief and detailed descriptions illustrated in Tab. B and respectively. We randomly selected questions and corresponding answers from these to construct Q&A pairs, enhancing the model's ability to engage meaningful dialogue based on the visual content. For continuous dialogues and complex reasoning, we used dialogues generated by G.T-3.. wwt.5-ttuooo00 0125. For instruction reconstruction, the commands are shown Tab. [5] which aim have the model generate detailed descriptions of visual stimuli using concise expressions. For concept localization, the commands are shown in Tab. used to extract concepts mentioned the prompts and visualize the model's attention using grad-
35
"Describe the image concisely."
PProvdde brief description of the given image."
"Offer succinct explanation of the picture presented."
"Summarize the visual content the image."
"Provide brief description of the image."
"Descrihe the image briefly."
"Summarize the image."
"Give a short and clear explanation of the subsequent image."
"Share concise interpretation of the image provided.
"Present a compact description of the photoš key features."
"Relay brief, clear account of the picture shown."
"Render a clear and concise summary of the photo."
"Write a but informative summary of the picture."
"Create a compact narrative representing the image presented."
Table 3: The list of instructions for brief description.
B fMRI Data Preprocessing
We initially preprocess the fMRI data to ensure consistency. Specifically, we resize the data to uniform dimensions using trilinear interpolation. The BOLD signal dimensions for subj01 are used as the standard, set at 83 x 104 x 81. After applying zero-padding to the edges, we divide the data into 14 14 patches preserve local information. We then employ a mask created from the union of the "nsdgeneral" regions across all subjects in the NSD dataset, which helps in eliminating information unrelated the visual stimuli. This process results in data formatted as N C, where N is the number of retained patches, and C represents the size of each patch.
During the training of feature extractors, we enhance data variability by applying MixUp 4 to different fMRI responses from the same subject for the same visual stimuli. MixUp coefficients are generated using a uniform distribution. with the mixing ratio UO0, 1). blend features from different trials. This technique aids in developing robust model by exposing it to interpolated data points, fostering generalization across varied neural responses.
C Architecture and Training Details
CLIP VIT-L/14 14] and Autoencoder KL. 15 were used as feature extractors for images, aligning with fMRT data. For the fMRI data, we employed a 16-layer Transformer Encoder (28 with hidden size of 768 to extract features, using the class token from the last layer the output. Two
15

---

Describe the following image detail.".
Provide detailed description of the given image.",
"ive an elaborate explanation
image you see.".
Share comprehensive rundown of the presented image.",
"Offer detailed description of image.".
"Descrihe the image in detail.",
"Offer a thorough analysis image.".
Prrvvide detailed explanation of the subsequent image.",
"Explain various aspects image before you.",
"Clarify the contents of the displayed image with great detail.".
"Characterize image using well-detailed description.",
"Break down the elements of the image in detailed manner.",
"Walk through the important details the image.".
PPortray the image with a rich, descriptive narrative.",
"Narrate contents the image with precision.".
Anaayye the image in comprehensive and detailed manner.",
"Illustrate the image through descriptive explanation.",
"Explain the image in detail.",
"Examine the image closely and share details.".
"Write exhaustive depiction of the given image.".
Table The list of instructions for detailed description.
"Provide the corresponding Stable Diffusion prompts for the image.".
Table 5: The list of instructions for instruction reconstruction.
"Locating the concept of "<ooject>",
Table The list of instructions for concept localization.
two-layer perceptrons with hidden dimension of 1024, fw. and fw., were used align with CLIP and VAE features, respectively, with the hyperparameter a set to 1/64, The learning rate was set 10-4, training for 30 epochs. For multimodal interaction, the aforementioned Transformer Encoder was frozen, using its second-to-last layer's 's hidden states as the fMRI token, processed through a two-layer perceptron fwy to interact with Llama-3-8B (44) Training was divided into two stages: initially, the I.I. was frozen, tuning only fay, and in the second stage, both the LLM and fw. were fine-tuned simultancously. The learning rate was set to x 10 training for one epoch. In the visual reconstruction phase, the LLM called upon UnCLIP-2 (20) for visual reconstruction, with the hyperparameter set to 0.93. For concept location, the LLM first extracted keywords and then localized them using GradCAM 35), visualizing the results.
All experiments were conducted on server equipped with NVIDIA 1.00 GPUs, each with 80 GB of memory. Training of the feature extractor was performed on single GPU with batch size set to 32, and the training duration across subjects was approximately 8 hours. For fine-tuning with LLMs, all 8 GPUs were utilized. During the phase where only fw, was fine-tuned, the batch size was set to 32, and the training time was about 4 hours. the phase where L.LMs were unfrozen for joint fine-tuning, the batch size adjusted 24, extending the training time to approximately 36 hours.
16

---

C.1 Metries for Visual Reconstruction
For evaluating visual decoding performance, we adhere to the established suite of eight metrics, commonly utilized in the field [ 9]. The metrics are divided into two categories: low-level and high-level. Low-level metrics include Pixelwise Correlation (PixCorr) and Structural Similarity
Index Metric (SSIM) [45]. well as Alex and AlexNet(5) [46], which measure the fidelity of reconstructed images against ground truth. High-level metrics comprise Inception [47], CLIP [14),
EfficientNet-B (EffNet-B) [48), and SwAV-ResNer50 (SwAV) 44 which evaluate the semantic
accuracy of the reconstructions.
Following protocols from previous research IT] we downsampled the generated images from resolution of 512 x 512 to 425 x 425, which matches the resolution of ground truth images in the NSD Dataset. This resolution adjustment was specifically for PixCorr and SSIM evaluations. For other metrics, the generated images were processed according to the input requirements of each respective model.
Two-way identification tests were conducted following the methodology of Ozcelik and Van- Rullen [42]. For each model, we calculated the Pearson correlation between embeddings of the ground truth image and its reconstruction, as well as between the ground truth image and another random reconstruction from the test set. A test was marked as correct if the correlation with the ground truth was higher than with the unrelated reconstruction. Performance for each test sample was averaged over all possible pairwise comparisons with the other 981 reconstructions to eliminate any bias from random selection. This resulted 982 averaged percentage correct outputs, which were then averaged to derive the final metrics presented in Tab
In addition to the established metrics, we introduced a testing protocol by utilizing only the first fMRI record for each visual stimulus to construct single-trial test [ This approach presents a more stringent and practical challenge, reflecting a scenario closer to real-world applications where each neural response is unique and not averaged over multiple instances.
D
More Results
D.1
Visual Reconstruction
Fig. 10 shows some randomly selected visual reconstruction results of different subjects under the single-trial condition. Consistent with the results in the main article, our method produces consistent visual reconstruction results across different subjects. This shows that method has good generalization performance across different subjects. But there are also some erroneous reconstruction results.
Table 7: Performance comparison on visual reconstruction tasks when using different LLMs and different instructions. The underline indicates the best result under the same conditions.
Method
Instruction
Low-l Level
High-L Level CLIP EffNettBB SwAV +
PixCorr
SSIM 1 AlexNet(2) AlexNet(5)
Inception
w/o LLM
.263
.369
92.0%
97.1%
94.2%
96.1%
.680
.328
w/ Vicuna- 13B
brietly descriptions
.259
.351
91.5%
96.6%
95.1%
96.7%
.641
.337
w/ Vicuna- 13B
instruction reconstruction
.257
.361
92.1%
96.9%
96.2%
97.19
.628
.331
L-ama-3-88
briefly descriptions
.354
92.7%
96.8%
96.4%
97.0%
.637
Lluma-3-8B instruction reconstruction
.265
.357
93.1%
97.1%
96.8%
97.5%
.633
Tab. 7 shows the performance comparison of visual reconstruction tasks when using different large language models (LLMs) and different generation instructions. We compared the two models Vicuna- 13B [50] and Llama-3-8B 44). The results indicate that our method is compatible with various LLMs and performs better in visual reconstruction tasks when using more powerful LLMs. When using Llama-3-8B, the best performance was achieved across the majority of metrics. Additionally, we explored the impact of different generation instructions on the visual reconstruction task. The results show that using instructions specifically designed for visual reconstruction can improve performance of the visual reconstruction task. We believe this is because these instructions enable the I.LMs to generate more detailed deseriptions visual stimuli, thereby enhancing the performance of the visual reconstruction task.
17

---

Visua Stimulus
Sub
Subjo2
Subjo3
Sub
Visual Stimulus
SubiC2
Subj03
Subj04
Figure 10: More visual reconstruction results for different subjects under single-trial conditions.
D.2
Language Interaction
We show some examples of multimodal interactions from IMRI data, shown in Fig. II The model conducted multimodal interactions based on fMRI signals and generated different forms of dialogue, including brief descriptions, detailed descriptions, and complex reasoning.
18

---

Oetaled Deverition
chaggote Desorite
feclosirg Inage
peacefully lath foald,
tail
scattared
prau taakgwas, cries cddligg soke von.. cams sarping calors, with led iroen stote, capasitie cantono
instroment
Free
Bnrf Restiatio
the sovittation cren dirld
Hatte positioes
C: dragerin Describe the Irage concisely.
seets spread aut acrous rather closeyy Sopathar, left siee others seatrened towedd their indicating more greeting behentor rether grastre.
A: herd standing field
Detalled Desgrigtion
shitee boanke bad. head peeting from undemeats alavat, illanireted pirtle mootie centrast wekiing faomy welter. The avenall stenddeen comntie, the exteylng
Brif Pestantin
cimageo Why the puvitioned ifie berch?
@: draggein Describe the Irage concisely.
At A cat is sitting wooden bench.
bench.
At The is positioned with its head and eyes lying confortablly the
Detalled
Desorite fellowing octail.
cave pallitiion Truards boe Tragg matteatie plaaed right appoors to wetlvved positien, vepeiting rmast wetting The rower io slock
entertateent witivity The
treffeestntioo
iinagesin Descrite the image concissly.
person holding reaite control In
person witching contrulling centrol. they Sesturely sestuity.
setting with blarket relaved
Detalled Descrigtion
cangg Desorite the fellowing suage detail.
covered
the tatle. seoms be rocused oe the delictous treats of donuts and ice crean being ercoyed by the person
holeiig doruts their hand. Meststanslly,
cinagosin Describe image concisoly.
chnogerin
At The nan in the inage is holding a donut and senc napkins. No appears be saiting, possibly Inficating postitiv expression.
holding donut and napkin
Figure 11: Some examples of multimodal interactions from fMRI data. The images are for reference only. The model performs forms of dialogue, including brief descriptions, detailed descriptions, and complex reasoning.
interaction based on fMRI signals and generates different
D.3 Concept Localization
In the section, explored the neural correlates of visual stimuli by mapping the captions derived from the visual content directly onto brain signals. This process involved using GradCAM [35] to generate heatmaps that visually represent the regions of the brain activated in response to specific elements of the visual stimuli. These heatmaps provide a compelling visualization of how different concepts associated with the images are processed across various areas of the brain.
Fig. 12 displays heatmaps that localize the brain activity corresponding to the captions of visual stimuli. These images are crucial for understanding the distribution and intensity of neural responses as they relate to the cognitive processing of visual information. By analyzing these heatmaps, we can infer which areas of the brain are most involved in the interpretation and semantic processing of the stimuli, providing insights into the underlying mechanisms of visual perception and cognitive response.
The localization of these concepts within the brain not only aids in validating theoretical models of brain function but also enhances our understanding of the cognitive processes involved in visual perception. Such detailed mappings are instrumental in advancing our knowledge of the brain's architecture and its functional connectivity in response to complex stimuli.
19

---

Visual Stmulus
Visual Stmulus
Responss Caption
Stimulus
the
0.0
3.0
Figure 12: Heatmaps illustrating brain regions activated by specific captions derived from visual stimuli, demonstrating the spatial distribution of neural responses.
E
Limitations
While our study introduces several innovative approaches to the decoding of non-invasive brain recordings and extends the capabilities of visual reconstruction using advanced computational models, there are several limitations that should be acknowledged:
Generalization across Diverse Populations: Our method was validated primarily using the Natural Scenes Dataset (NSD), which consists of data from a limited number of subjects. Although we demonstrate robustness across these subjects, the generalizability of our findings to broader pop- ulations remains an area for further investigation. Differences neural anatomy and functional organization across individuals that are not represented in the NSD could affect the efficacy and accuracy of our model in wider applications.
Computational Complexity and Resource Requirements: The implementation of our framework, particularly the integration with Large Language Models (LLMs) and advanced image processing techniques like VIT3D, requires substantial computational resources. This might restrict the utility of our approach in environments with limited computational capacity or in real-time applications where rapid processing crucial.
20

---

Challenges in Real-World Application: While the single-trial test setting introduced in our study adds a layer of practical relevance by evaluating model performance a more realistic scenario, also presents challenges. The variability in single-trial fMRI responses, which can be influenced by numerous uncontrollable factors such as minor head movements or physiological fluctuations, may lead to inconsistencies in decoding accuracy. This variability emphasizes the need for further refinement noise reduction and signal processing techniques.
The development and application of brain decoding technologies raise ethical questions, particularly concerning privacy and consent. As our methods advance and poten- tially become capable of decoding more detailed and sensitive information from brain data, ensuring ethical deployment and the protection of participant data becomes paramount.
Ethical Considerations:
These limitations highlight the need for continuous improvement and careful consideration in the deployment of brain decoding technologies. Addressing these challenges through further research and development will be crucial for realizing the full potential of non-invasive brain decoding in both scientific research and clinical applications.
21

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results
be expected to generalize other settings.
is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed in Appendix
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
22

---

Justification:
The manuscript does not contain theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental Result Reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: In Appendices A
B and C we describe in detail the settings required to reproduce the experiment, and for the datasets and key code that will be released with this manuscript, we have provided it in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
23

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: Code and language extensions for the NSD dataset are provided in the supple- mentary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please the NeurlPS code and data submission guidelines (https://nips.. cc/
public/ /uidds/uusssse/eee Codee aasioPooliry for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to reproduce the results. See the NeurIPS code and data submission guidelines (https
hpppblccclllllpp publi c/ des/ Code bm ss onPol icy for more details. / //nips.com cc/
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/eetails
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Detailed training details and evaluation metrics are provided in Appendices B and C
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: The results of the manuscript do not include error bars, mainly for the following reasons: 1. In order to make fair comparison with other methods, other literature in this field does not report error bars. 2. Multiple experiments involving LLMs are computationally expensive. 3. The results of this manuscript have been verified different subjects, and the results are consistent.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
24

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative eπτor rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Detailed hardware facilities and computational overhead are provided in Appendix B
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification: This manuscript does not involve direct human subjects; all data used from publicly available datasets that comply with the ethical standards of NIPS.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., s if there is special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Detailed social impact is discussed in the Conclusion and Appendix Guidelines:
The answer NA means that there is no societal impact of the work performed.
25

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
There is no relevant risk.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g.. code, data, models), used in the paper, properly credited and the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification:
We do cite all the existing assets in our paper as well as in our codebase.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
26

---

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paaperswi haperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: The paper does not release new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/codeemoodel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
27

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
28


Reviews:
[{"review_id": "V7AX7xgOZ8", "reviewer": "Reviewer_fArY", "summary": "- This model focuses on the task of reconstructing image stimuli from fMRI readings\n- Instead of training a subject specific model, a subject-generic model is trained\n- The model is built around a pre-trained LLM core, which is finetuned to take the fMRI as input and then:\n  - engage in dialogue about the image stimulus\n  - reconstruct the image stimuli by generating a language prompt which is then passed to UnCLIP\n  - create a heatmap of activations in the brain associated with a given concept", "questions": "- line 293 --- how are concepts \"extracted\"?\n- line \"Finally, patches containing non-task-relevant information are filtered out\" -- how are the masked portions chosen?\n- line 256-258: I think there is a typo somewhere. I'm not sure what the sentence is supposed to say.\n- Fig 1: From the figure, it seems that one can query the LM for concept localization in the brain? But where is that described? What do annotations for that dataset look like?\n- A related question: line 533: what does it mean to localize the results using GradCAM? Can you describe how that was done?\n- Fig 5 and 6: Equation (4) says that $\\beta=1$ corresponds with all noise input. But increasing noise seems to correspond with better image quality?\n- Line 484: Typo: \"Language Extension\"\n- Table 2: what is `C_Subj`?", "limitations": "Limitations and potential negative societal impact are discussed in the appendix.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The multi-subject aspect of the training appears novel\n- The use of an LLM for image reconstruction and dialogue about the stimulus also appears novel\n- The image reconstructions are of comparable or better quality than existing approaches\n- Ultimately, I think the evaluation could be improved (see Weaknesses), but given the novelty of the approach (using an LLM to do decoding and reconstruction), I argue for acceptance.", "weaknesses": "- It's unclear how much the image reconstructions reflect visual processing in the brain. It would be good to do an ablation to see how much the reconstruction depends on $\\hat{z}_c$. What if $\\hat{z}_c$ were replaced with random noise in the same way that an ablation was done on $\\hat{z}_v$? Basically, I am curious about how much decoding can be done from the prompt $a_r$ only.\n- In the same vein, it would be interesting to do an experiment where the generated prompt $a_r$ is given to a generative model that does not require $z_c$ and $z_v$, i.e., a model that can take a text prompt and generate an image.\n- It's claimed that the model can engage in complex dialogue about the semantic content of the stimuli, but from the example prompts, it seems that many of the answers to the dialogue questions may be available in the prompt. For example, figure 11 seems to suggest that many of the complex reasoning prompts contain enough information to answer the question in the prompt. E.g. ``What is the distribution of the cows in the field based on the description and their positions in the image.'' A good ablation test would be to perform the complex reasoning task without the fMRI inputs.\n- Some technical details are hard to follow, especially with regards to concept localization (see questions).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your detailed and insightful review. Below, we provide a brief summary of your questions and our responses:\n1. **Impact of $\\hat{z}_c$ on image reconstructions: How much does the reconstruction depend on $\\hat{z}_c$? What if $\\hat{z}_c$ were replaced with random noise?**\n\n   We conducted an ablation study where $\\hat{z}_c$ was replaced with random noise, and another experiment where $\\hat{z}_c$ was set to zero. The results, summarized in **Table R1** of the attachment, indicate that both $\\hat{z}_c$ and $\\hat{z}_v$ significantly contribute to reconstruction quality. Using $a_r$ alone, derived solely from LLM prompts, achieved some level of reconstruction quality, particularly maintaining high-level semantic information. However, it affected the performance of lower-level image details more significantly. We observed that setting $\\hat{z}_c$ to zero yielded better results than replacing it with random noise, suggesting that random fMRI embeddings might negatively impact reconstruction quality.\n2. **Experiment with text prompt only: Can a generative model create an image from $a_r$ without using $z_c$ and $z_v$?**\n\n    We conducted this experiment using Stable Diffusion 2-1 [1], which generates images from text prompts alone. The performance, detailed in **Table R1**, demonstrates that even without directly using fMRI Feature Extractor, text prompts $a_r$ can achieve notable reconstruction quality. This highlights that LLMs can effectively interpret brain signals, and this method avoids the negative impact of missing fMRI embeddings on reconstruction quality.\n3. **Does the prompt contain enough information to answer complex reasoning questions without fMRI inputs?**\n\n    Your concern is valid. In multimodal scenarios, information leakage through the question is possible. Thus, we performed an ablation experiment by removing the `<image>` tag from the question $q$, effectively testing without fMRI embeddings. The results, shown in **Table R2** of the attachment, indicate that for tasks like Brain Caption and Detail Description, where contextual information is not leaked through the question, removing fMRI embeddings results in random-level performance, validating the effectiveness of LLMs in extracting brain signals. For Complex Reasoning tasks, while some scene information is leaked through the question $q$, the inclusion of fMRI inputs significantly enhances response quality and accuracy, confirming the effectiveness of our method.\n4. **How are concepts \"extracted\"?**\n\n    As shown in Table 6 on page 15, we fine-tuned the LLM to recognize concept localization instructions. Upon identifying such instructions, the LLM calls functions to perform concept localization. Specifically, the `<concept>` is input into the CLIP Text Encoder. Our fMRI Feature Extractor, aligned through training, uses the CLIP Embedding of `<concept>` and applies GradCAM for concept localization and visualization, identifying the brain regions corresponding to the textual concept.\n5. **How are the masked portions chosen?**\n\n    We used the `nsdgeneral` labels from the Natural Scenes Dataset (NSD) to filter out non-task-relevant brain regions. Specifically, the union of `nsdgeneral` regions for each subject in the NSD dataset was considered task-relevant, and other regions were removed. Detailed processing of the dataset is described on line 473, page 13 appendix.\n6. **Line 256-258: I think there is a typo somewhere. I'm not sure what the sentence is supposed to say.**\n\n    Apologies for the confusion. `w/o ViT3D` refers to not using the 3D structure-preserving fMRI preprocessing method described in section 3.1 on page 4 but instead flattening the fMRI data and patching it as in other literature to maintain the same fMRI feature extractor structure. We will clarify this in the revised manuscript. Thank you for your correction.\n7. **Fig 1: From the figure, it seems that one can query the LM for concept localization in the brain? But where is that described? What do annotations for that dataset look like?**\n\n    Yes, in our method, the LLM can be queried for concept localization. The specific implementation is detailed in our response to question 4. The method is described on lines 211-214, page 6. Concept localization experiments are discussed on page 9, section 4.4, and in the appendix, section D.3 on page 18. The dataset annotations used are from the NSD, which includes visual stimuli images and fMRI data from subjects viewing these images, along with corresponding COCO captions.\n8. **What does it mean to localize the results using GradCAM? Can you describe how that was done?**\n\n    We aim to use GradCAM to visualize the contributions of different natural language representations of concepts within fMRI signals. This involves querying the LLM for concept localization as described in our response to question 4.\n9.  **Increasing noise seems to correspond with better image quality?**\n\n    Equation (4) describes the visual stimulus reconstruction process using UnCLIP. Specifically, $\\hat{z}_c$ represents fMRI features aligned with CLIP, and $a_r$ represents LLM-generated captions. Noise is added to the VAE features used to initialize the diffusion process. Due to the nature of the diffusion process, insufficient noise can destabilize the process and affect the injection of high-level semantic information $\\hat{z}_c$ and $a_r$, impacting reconstruction quality. Adequate noise helps stabilize the diffusion process, improving reconstruction quality. This is discussed in Figure 5 on page 8.\n10. **Typo: \"Language Extension\"**\n\n    The typo has been corrected.\n11. **What is C_Subj?**\n\n    `C_Subj` refers to cross-subject training, where the model is trained on data from multiple subjects simultaneously, rather than training separate models for each subject as done in other studies.\n\nWe hope these clarifications address your concerns and enhance the quality of our work. Thank you for your constructive feedback.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OjMIINYiDe", "reviewer": "Reviewer_h32m", "summary": "The paper introduces a new, subject-agnostic visual reconstruction pipeline. They introduce a way to integrate across fMRI readings from different subjects and enhance their integration using LLMs. Through this integration they see a consistent improvement of high level semantic feature baselines in their reconstruction.", "questions": "* What does fine-tuned model refer to in line 204? Is it the LLM (just making sure)\n* When you use BLIP2 generated captions, are you passing the original NSD image and generating a caption? Is this in comparison to your LLM that uses an fMRI embedding in place of [image]? I think this was confusing. \n* Although GradCAM is easily interpretable, I do wonder if this would benefit from something more recent for interpretability like [1]. \n* Could MindEye2 (https://medarc-ai.github.io/mindeye2/) be used as a baseline? \n\n[1] Materzynska et. al. Disentangling visual and written concepts in CLIP. CVPR 2022.", "limitations": "* Limitations are addressed but may want to consider toning down some of the claims about subject variability (see weaknesses).", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "* An interesting new approach to align fMRI feature extraction with ViTs and LLMs\n* Brain captioning with LLMs is highly novel. The results are promising in Table 1.\n* Promising results on NSD reconstruction.\n* Interesting set of analyses using GradCAM.", "weaknesses": "* Concerns about preprocessing and claims about cross-subject analysis\n    * The authors propose a trilinear interpolation to combine BOLD signals across patches across subjects. To me, this makes many assumptions that may not be true or might miss several problems. The authors need to address these questions to substantiate the claim of removing subject-specific modeling or using some kind of mixed subject pretraining to combine subjects. \n    * For example, this method ignores inter-subject variability such as anatomical differences or functional differences, which can vary among individual subjects. \n    * The interpolation makes a linear assumption on BOLD responses which may not be true. I realize this will follow with alignment to a ViT later but this imposes strong assumptions about BOLD signal space. \n    * My main concern here is whether this kind of preprocessing will generalize to modeling over new subjects. Is this approach applicable to general subject populations? Could the authors add some clarification and expand on this point?\n* LLM Interaction details\n    * My understanding is that there is a finetuning objective when using the LLM interaction. I think section 3.4 was cut off somehow and this wasn’t very clear to me.\n    * Some details were missing making Table 1 a bit difficult to understand. See questions", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your insightful comments. Your feedback has helped us enhance the quality of our manuscript. Below, we will provide detailed responses to your comments.\n### **Concerns about Preprocessing and Inter-Subject Variability:**\nWe appreciate your concern regarding inter-subject variability. Developing a universal model for all subjects avoids the need for separate models for each brain, reducing computational and storage costs. Our cross-subject approach, validated by concurrent studies at CVPR 2024, ICML 2024, and ECCV 2024 (References [3-5]), uses a voxel patching technique to capture essential brain structures without additional subject-specific parameters. This enhances robustness, generalizability, and spatial integrity, resulting in superior performance over subject-specific models (Table 2, page 7). By using a trilinear interpolation method, we ensure effective alignment of cross-subject characteristics and achieve high-quality reconstruction and interpretability at low computational costs. Our model's ability to generalize without extra parameters for each subject sets it apart from existing approaches and is a critical step toward universal applicability.\n1. **Trilinear Interpolation and Assumptions:**\n   \n    We acknowledge your concerns regarding the assumptions underlying our use of trilinear interpolation. To address this, we compared various interpolation methods, including `nearest`, `area`, and `nearest-exact`, finding minimal impact on performance (detailed in **Table R3**). Trilinear interpolation effectively manages inter-subject variability and supports efficient cross-subject data integration while preserving essential spatial structures, which is critical for robust and generalizable model performance.\n2. **Anatomical and Functional Differences:**\n   \n    Our approach includes a voxel patching technique that captures local anatomical and functional differences, which are vital for accurately modeling BOLD signals. This method significantly enhances the performance over models that do not maintain these structures (results shown in Table 2).\n3. **Linear Assumption on BOLD Responses:**\n   \n    Although a linear assumption is involved, our results confirm that trilinear interpolation preserves the spatial integrity of BOLD signals. This integrity is crucial for effective alignment with the ViT structure and thus enhances our model's performance, especially when compared with models lacking the ViT3D structure (`w/o C_Subj & ViT3D`).\n4. **Generalization to New Subjects:**\n   \n    Our model has shown significant superiority over subject-specific models (Table 2), indicating robust generalization capabilities that make it highly suitable for broader subject populations. This advantage underscores the scalability and robustness of our approach, which does not require tailoring to individual subjects, thus simplifying the application to a diverse range of subjects. We intend to further validate our method on a larger dataset and have enriched our manuscript with discussions that clarify the universal applicability and potential of our approach to extend to more generalized settings.\n### **LLM Interaction details:**\n1. **Fine-Tuning Objective:**\n   \n    The fine-tuning of the LLM using fMRI embeddings aligns the penultimate hidden state of the fMRI feature extractor with the LLM’s text embeddings. This integration allows the LLM to interpret fMRI data as natural language inputs, facilitating complex reasoning and language-based tasks. The fine-tuning occurs in two phases: initially training only the MLP between the fMRI feature extractor and the LLM (with the LLM frozen) and subsequently training both components together. Detailed process and results are documented on lines 234-241 and in Appendix A.\n### **Responses to Questions:**\n1. **What does fine-tuned model refer to in line 204? Is it the LLM (just making sure).**\n   The 'fine-tuned model' mentioned refers specifically to the LLM. This model has been adapted to align with fMRI embeddings, enhancing our ability to interpret complex neural data as described in Table 7 on page 16.\n2. **BLIP2-Generated Captions:**\n   \n    Regarding the use of BLIP2-generated captions, we clarify that we utilize the original NSD images as input to BLIP2 to generate target captions. These captions are then utilized to compare with those generated by LLMs, which uses fMRI embeddings instead of direct image inputs. This comparison is vital for assessing our model's capability in natural language tasks, particularly considering the differences brought by the cropping process in the NSD images, as illustrated in Figure 9.\n3. **Interpretability Methods:**\n   \n    Thank you for the suggestion regarding interpretability methods. Materzynska et al.[2] provide valuable insights for visual semantic interpretation, our focus on localizing textual information within brain signals benefits from the established efficacy of GradCAM. We employ this method to pinpoint text concepts in neural data. Nonetheless, we are open to integrating more recent techniques to potentially enhance our model's interpretability. Our manuscript includes experiments, specifically in Figures 7-8 and 12, that demonstrate the robustness and effectiveness of our approach in localizing these concepts.\n4. **MindEye2 as Baseline:**\n   \n    Thank you for suggesting MindEye2 as a baseline. We have conducted a comparative analysis between MindEye2 and our method, presented in **Table R3** of the attachment. Our method outperforms MindEye2 in most metrics, especially for models not fine-tuned on specific subjects. Even compared to MindEye2 models fine-tuned on specific subjects, our method shows superior performance on high-level semantic metrics. This underscores our framework's scalability and effectiveness across subjects.\n\nWe trust that the above responses comprehensively address your queries. Please inform us of any additional details you would require.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jF9KrdDwWg", "reviewer": "Reviewer_k95k", "summary": "This paper presents an innovative framework that leverages Vision Transformer 3D (ViT3D) to integrate 3D brain structures with visual semantics, enhancing visual reconstruction and language interaction from fMRI data. By aligning fMRI features with visual embeddings through a unified feature extractor and integrating with LLMs, the framework enhances decoding capabilities, enabling tasks such as brain captioning, complex reasoning, concept localization, and visual reconstruction. The framework demonstrates exceptional performance and holds significant potential for applications in neuroscience and human-computer interaction.", "questions": "I am curious if the method of combining low-level and high-level features in eq4 is the most effective. While other works also use this approach, it seems to force the model to trade off between the two types of features. Are there potentially better methods?\nIn visual reconstruction, the method with LLM appears effective only when the beta value is low. Does it overlap in function with the CLIP embedding?", "limitations": "The authors discuss the limitations well in the appendix.", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Provides a multi-task visual neural decoding framework that is simpler and more elegant compared to previous methods, offering a robust example for integrating visual and neural signals with unified modeling. Introduces a method for precise localization of language-based concepts within brain signals simply using the GradCAM method.", "weaknesses": "The Dual-Stream fMRI Feature Extractor is already a common method in the field, but the combination of the two features appears suboptimal. The authors thoroughly discuss the trade-off between low-level and high-level features. However, based on the results in fig 4, the reconstructed images’ low-level features are inferior to previous methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your detailed and insightful review. Your positive feedback on the soundness, presentation, and contribution of our work is highly encouraging. We are pleased that you found our framework innovative and robust in integrating visual and neural signals. We will now list your concerns one by one and provide specific responses.\n\n1. **The Dual-Stream fMRI Feature Extractor is already a common method in the field, but the combination of the two features appears suboptimal. The authors thoroughly discuss the trade-off between low-level and high-level features. However, based on the results in fig 4, the reconstructed images’ low-level features are inferior to previous methods.**\n   \n    Thank you for highlighting this issue. While the dual-stream fMRI feature extractor is indeed common, one of our main innovations is maintaining the spatial structure of the fMRI signals while achieving high-quality cross-subject feature extraction. Our method integrates these features with Vision Transformer 3D (ViT3D) and LLMs. This integration aims to balance the trade-off between low-level and high-level features, which is a known challenge in the field.\n\n    In Figure 4a, the difference in decoding quality for low-level features can be attributed to our model being cross-subject, meaning a single model can process brain signals from different subjects without the need for subject-specific training. Other methods require training separate models for each subject and often include additional low-level features such as depth and color (as seen in Ref [6]). Our approach, while compatible with these methods, focuses on reconstructing high-level feature representations and cross-subject generalization, thereby achieving better overall performance in tasks requiring complex semantic understanding. This trade-off is crucial for advancing multi-task neural decoding. For Figure 4b, our method shows better reconstruction quality at different levels when compared to single visual stimulus reconstruction tasks. Detailed quantitative analysis can be found in Table 2 on page 7 of the manuscript.\n\n\n1. **I am curious if the method of combining low-level and high-level features in eq4 is the most effective. While other works also use this approach, it seems to force the model to trade off between the two types of features. Are there potentially better methods? In visual reconstruction, the method with LLM appears effective only when the beta value is low. Does it overlap in function with the CLIP embedding?**\n   \n    Thank you for your insightful question. The trade-off between different levels of features is indeed a challenge. We have systematically discussed how to balance these features by regulating the initialization noise in the Diffusion latent space in Figure 5. While our current method shows significant benefits, we recognize it may not be the optimal approach. We are actively exploring alternative fusion techniques, such as hierarchical feature integration and dynamic weighting mechanisms, to alleviate this trade-off.\n\n    Regarding the beta value, our experiments indicate that the method using LLM is more effective at lower beta values. This suggests that a low \\( \\beta \\) value better integrates high-level semantic information. Our method aligns brain signals directly with the Diffusion latent space, and if \\( \\beta \\) is too low, it causes instability in the diffusion process due to the large discrepancy between the latent space and Gaussian distribution. In this case, LLM provides additional high-level semantic information, stabilizing the reconstruction through extra guidance.\n    \n    While CLIP embedding indeed overlaps in function with LLM by providing high-level semantic information during the reconstruction process, combining both can enhance reconstruction quality, as illustrated in Figure 5 on page 8 of the manuscript and **Table R1** of the attachment.\n\nWe are committed to continuously refining our methodology and appreciate your valuable feedback, which has been instrumental in enhancing the quality and clarity of our manuscript. Please let us know if further details are required or if additional modifications are needed.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "gdlXJnEbb2", "reviewer": "Reviewer_4C2X", "summary": "This paper proposes a framework that integrates 3D brain structures with visual semantics using a Vision Transformer 3D. By aligning fMRI features with multiple levels of visual embeddings, it eliminates the need for subject-specific models and allows extraction from single-trial data. The extractor consolidates multi-level visual features into one network, simplifying integration with LLMs. The topic is intriguing, and the proposed method offers practical applications.\n\nMajor concerns and minor comments include:\n\n1. Benchmarking with Real-World Datasets: As a novel machine learning approach, the performance of the proposed method should be benchmarked with more real-world fMRI datasets to evaluate the generalizability of the results. The current study may be limited in scope or sample size, and using a wider variety of datasets will help demonstrate the robustness and scalability of the approach across different subjects and conditions. It would be beneficial to include datasets with diverse characteristics, such as different brain regions, tasks, and populations, to ensure comprehensive evaluation.\n\n2. Discussion on Advantages and Disadvantages: The author(s) should discuss the advantages and disadvantages of the proposed method in the field of neuroscience and brain decoding. This discussion should include a comparison with existing approaches, highlighting the unique contributions and potential limitations of the new method. Additionally, insights into the interpretability and explainability of the proposed method would be valuable. For instance, how does this method enhance our understanding of brain activity patterns? Are there any trade-offs between model complexity and interpretability? Addressing these questions will provide a clearer picture of the method’s potential impact and areas for improvement.\n\n3. Details on Cross-Validation: Please provide more details about the cross-validation used in the empirical studies, perhaps in Section 4. It is essential to specify the type of cross-validation technique employed (e.g., k-fold, leave-one-out) and the rationale behind its selection. Detailed information on the partitioning of the data, the number of folds, and any stratification strategies used will help in understanding the robustness of the validation process. Additionally, discussing the metrics used for evaluation and how they were computed across different folds will add clarity to the reported results.\n\n4. Pseudocode for the Proposed Method: The proposed method can be summarized in the form of pseudocode (algorithm). Providing a step-by-step algorithmic representation will make the methodology more transparent and easier to reproduce. The pseudocode should outline the key steps involved in data preprocessing, feature extraction, model training, and inference. Including comments within the pseudocode to explain the purpose of each step and any critical hyperparameters or configurations will further enhance understanding.\n\n5. Minor Typos and Grammar Mistakes: There are some minor typos and grammatical mistakes throughout the paper. A thorough proofreading and editing process is recommended to improve the overall readability and professionalism of the manuscript.", "questions": "Please see the Summary section", "limitations": "Please see the Summary section", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Please see the Summary section", "weaknesses": "Please see the Summary section", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for the thorough review and positive feedback on our work. Your comments are invaluable for the improvement of our method and the revision of the manuscript. Let us address each individual comment/question below.\n\n\n1. **Benchmarking with Real-World Datasets:**\n   \n    We fully recognize the importance of demonstrating our model's robustness across diverse datasets. The Natural Scenes Dataset (NSD) that we utilized is the largest available for natural scene visual stimulation and includes data from multiple subjects. While there are other datasets, their smaller size and variations in acquisition equipment often limit their suitability for benchmarking our method comprehensively. Our framework is specifically designed to be scalable across different subjects without requiring subject-specific parameters, a feature that greatly enhances its potential for broader application.\n    This scalability is a fundamental strength of our approach, as demonstrated by consistent performance across various subjects, which is detailed in Section 4.2 and Appendix A of the manuscript. We are actively planning to extend our evaluations to additional datasets in the future, aiming to further validate and refine our model’s capabilities across an even wider array of conditions and populations.\n\n\n2. **Discussion on Advantages and Disadvantages:**\n   \n    Thank you for prompting a deeper exploration of the strengths and potential limitations of our approach. We have detailed our discussions on these aspects in the Related Work section of the manuscript. Our framework's ability to efficiently reconstruct visual stimuli from a single input across diverse subjects is comprehensively evaluated, with results available in Table 2 on page 7 and Figure 4 on page 8.\n    Additionally, the integration of a large language model has expanded our system's functionality, enabling it to perform a range of natural language processing tasks, from simple descriptions to complex reasoning. This not only demonstrates the versatility of our approach but also ensures its adaptability to various research and clinical settings.\n    In addressing interpretability, we utilize Gradient-weighted Class Activation Mapping (Grad-CAM) to localize semantic concepts within specific brain regions effectively. This technique enhances our understanding of how cognitive functions correlate with neural activity, allowing us to map textual information directly to brain signals. Our approach's accuracy is further validated by ablation studies where the removal of concept-related brain signals confirms the efficacy of our localizations, as discussed extensively in Section 4.4.\n    Following your suggestion and insights from Reviewer k95k, we are exploring additional interpretability techniques and plan further experiments to enhance our model’s explainability without compromising its effectiveness. We appreciate your insights and are committed to continuously refining our method to enhance its robustness and applicability.\n\n3. **Details on Cross-Validation:**\n   \n    We have detailed our dataset preprocessing and validation strategy in Section 4.1 on page 6 of the manuscript and the first three sections of the appendix. The NSD uses a standard 9:1 split for the training and validation sets, a common methodology that ensures a fair comparison with other methods. This split was determined during the dataset's compilation, where a fixed set of images shown to all subjects was designated as the validation set, allowing for consistent and comprehensive comparisons across studies. We appreciate your attention to this aspect of our work and are committed to ongoing improvements in our research methodologies.\n\n4. **Pseudocode for the Proposed Method:**\n   \n    To enhance clarity and reproducibility, we've included a pseudocode summary of our method in the appendix. Here's the markdown-formatted pseudocode:\n\n    ```markdown\n    ### Pseudocode for Neuro-Vision to Language Framework\n\n    - **Input**: fMRI data, Visual Stimuli\n    - **Output**: Visual reconstructions, Semantic interactions\n\n    **Procedure**:\n    1. **Preprocess fMRI Data**:\n        - Normalize and segment fMRI data to prepare it for feature extraction.\n    2. **Feature Extraction with Vision Transformer 3D (ViT3D)**:\n        - Apply ViT3D to the preprocessed patches to extract spatial feature embeddings.\n    3. **Align Features with Visual Embeddings**:\n        - Align fMRI features with visual embeddings from CLIP and VAE, optimizing with MSE loss.\n    4. **Integrate with Large Language Model (LLM)**:\n        - Feed aligned features into LLM for fine-tuning; generate natural language responses.\n    5. **Visual and Semantic Reconstruction**:\n        - Convert semantic embeddings back to visual space; perform concept localization and dialogue-based interactions.\n    ```\n    This pseudocode comprehensively outlines the steps from data preprocessing to final inference, detailing critical hyperparameters and methodological choices, such as feature alignment techniques and the integration of visual and language models. This structured algorithmic representation should facilitate researchers in replicating our approach and exploring its potential adaptations. Additionally, we have provided key code for all experiments in the supplementary materials, ensuring that other researchers can directly replicate and validate our results.\n\n5. **Minor Typos and Grammar Mistakes:**\n   \n    We have rigorously proofread the manuscript to correct all typographical and grammatical errors.\n\nWe thank you again for your insightful critique, which has significantly enhanced the quality and clarity of our research presentation. We remain receptive to further feedback and are prepared to refine our manuscript as needed to meet the expectations of the readership.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We sincerely thank you for your insightful feedback. We have carefully reviewed your comments and provided specific responses to each point raised. We appreciate the opportunity to discuss our research in more depth and clarify any concerns through these responses.

In the attached response document, we have conducted supplementary experiments to address the specific doubts raised, including:

- **Table R1**: Analyzes the contributions of $\hat{z}_c$ and $a_r$ in stimulus reconstruction, demonstrating the best performance when both are utilized together.
- **Table R2**: Discusses the performance drop in question-answering tasks without fMRI embeddings, underlining the essential role of these embeddings.
- **Table R3**: Evaluates different cross-subject alignment methods and compares our approach with MindEye2, showcasing our method's robustness and effectiveness.

---

The references cited in our specific responses include:

[1] Rombach, Robin, et al. "High-resolution image synthesis with latent diffusion models." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2022.  
[2] Materzynska, et al. "Disentangling visual and written concepts in CLIP." CVPR, 2022.  
[3] Wang, Shizun, et al. "Mindbridge: A cross-subject brain decoding framework." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2024.  
[4] Scotti, Paul Steven, et al. "MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data." Forty-first International Conference on Machine Learning, 2024.  
[5] Xia, Weihao, et al. "UMBRAE: Unified Multimodal Brain Decoding." European Conference on Computer Vision (ECCV), 2024.  
[6] Xia, Weihao, et al. "Dream: Visual decoding from reversing human visual system." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, 2024.

Meta-review: The paper proposes reconstructing image stimuli from fMRI, using a subject-generic model. This model is built around a pre-trained LLM core, which is finetuned to take the fMRI as input and then, it engages in dialogue about the image stimulus, reconstructs the image stimuli by generating a language prompt which is then passed to UnCLIP and create a heatmap of activations in the brain associated with a given concept.
The reviewers agreed that there was technical novelty in the paper, although it is yet-another-paper on NSD.
The authors delivered a convincing ablation study.
Some questions are left open such as whether the reconstruction reflects visual or semantic processing. The use of GradCam entails some limitations that would also have deserved a discussion.

Overall, after the rebuttal, the paper is found compelling, and there is a clear consensus for acceptance. On the other hand, it sounds like a successful attempt rather a strong conceptual contribution, and an oral presentation is not requested.
Decision: Accept (poster)