Seeing Through the Brain: New Insights from Decoding Visual Stimuli with fMRI
Authors: Zheng Huang
Enpei Zhang
Weikang Qiu
Yinghao Cai
Carl Yang
Elynn Chen
Xiang Zhang
Rex Ying
Dawei Zhou
Yujun Yan
Venue: ICLR 2026 Oral (2026)
Keywords: Neuroscience
Functional Magnetic Resonance Imaging
Image reconstruction
Reconstruction
Primary area: applications to neuroscience & cognitive science
TL;DR: We present PRISM, a framework to decode visual stimuli from fMRI with language model alignment

Abstract: Understanding how the brain encodes visual information is a central challenge in neuroscience and machine learning. A promising approach is to reconstruct visual stimuli—essentially images—from functional Magnetic Resonance Imaging (fMRI) signals. This involves two stages: transforming fMRI signals into a latent space and then using a pre-trained generative model to reconstruct images. The reconstruction quality depends on how similar the latent space is to the structure of neural activity and how well the generative model produces images from that space. Yet, it remains unclear which type of latent space best supports this transformation and how it should be organized to represent visual stimuli effectively.

We present two key findings. First, fMRI signals are more similar to the text space of a language model than to either a vision-based space or a joint text–image space. Second, text representations and the generative model should be adapted to capture the compositional nature of visual stimuli, including objects, their detailed attributes, and relationships. Building on these insights, we propose PRISM, a model that Projects fMRI sIgnals into a Structured text space as an interMediate representation for visual stimuli reconstruction. It includes an object-centric diffusion module that generates images by composing individual objects to reduce object detection errors, and an attribute/relationship search module that automatically identifies key attributes and relationships that best align with the neural activity.
Extensive experiments on real-world datasets demonstrate that our framework outperforms existing methods, achieving up to an 6% reduction in perceptual loss. These results highlight the importance of using structured text as an intermediate space to bridge fMRI signals and image reconstruction. Codes are available at https://github.com/GraphmindDartmouth/PRISM.

Published as a conference paper at ICLR 2026
SEEING THROUGH THE BRAIN: NEW INSIGHTS FROM DECODING VISUAL STIMULI WITH FMRI
Zheng Huang': Enpei Empei Zhang'; Weikann Qiu?, Yinghao Cai', Carl Yang", Elynn Chen', Xiang Zhang,, Rex Ying", Dawei Zhou', Yujun Yan
Dartmouth College Yale University EEmory University
New York University
UNC Charlotte VVirginia Tech
ABSTRACT
Understanding how the brain encodes visual information a central challenge neuroscience and machine learning. A promising approach is reconstruct vi- sual stimuli-essentiall images-fron functional Magnetic Resonance Imaging (fMRI) signals. This involves two stages: transforming fMRI signals into a latent space and then using a pre-trained generative model to reconstruct images. The reconstruction quality depends how similar the latent space is to the structure of neural activity and how well the generative model produces images from that space. Yet, it remains unclear which type of latent space best supports this trans- formation and how it should be organized to represent visual stimuli effectively.
We present two key findings. First, fMRI signals are more similar to the text space of a language model than to either vision- based space or a joint text image space. Second, text representations and the generative model should be adapted to capture the compositional nature of visual stimuli, including objects, their detailed attributes, and relationships. Building these insights, we propose PRISM, model that Projects IMRI sIgnals into a Structured text space as an inter\eediate representation for visual stimuli reconstruction. It includes an object-centric dif- fusion module that generates images by composing individual objects to reduce object detection errors, and an atrribute/relationshipp search module that automati- cally identifies key attributes and relationships that best align with the neural activ- ity. Extensive experiments on real-world datasets demonstrate that our framework outperforms existing methods, achieving up to an 6% reduction in perceptual loss. These results highlight the importance of using structured text as an intermediate space bridge IMRI signals and image reconstruction. Codes are available at
INTRODUCTION
Decoding visual stimuli from brain activity provides a unique lens into human perception (Naselaris et al., 2011 Haufe et al. 2014). A central approach uses fMRI signals- which measure neural activity through subjects Allen al. 2022 Chang et al. 2019 Luo et al. 2023). Recent advances in deep generative models have significantly improved these reconstructions, deepening our understanding of visual representation in the brain (Chen al.
responses to reconstruct the images perceived by
2023) and enabling applications in brain-computer interfaces
(Sitaram et al., 2008) and brain-driven content generation (Wang et al. 2024a Qiu et al. 2025).
FMRI-to- tmaaee reconstruction involves two stages: mapping fMRI signals into latent space and then generating images from that space. Its success depends on the similarity between the latent space and neural activity (alignment) and how well the generative model produces high-quality im- ages. While recent studies (Scotti et al.] [203] [2024; Mai et al.] (2024) focus on enhancing image quality using advanced generative models (Podell et al. 2023; Xu et al., 2023), alignment remains underexplored. Prior work often assumes that the latent space should match the modality of the stimuli, i.c., using vision model representations to reconstruct visual stimuli (Scotti et al., 2023 Wang et al. 2024c Xia al. 2024 Some studies incorporate auxiliary semantic information from
"Equal contribution. Correspondence aileng.haanggeee dartmouth.edu.

---

Published as a conference paper at ICLR 2026
language models (LMs) (Lin al. 2022 Quan et al. 2024). but still rely on vision-based represen- tations as the core latent space. In contrast, we question whether matching the modality of visual stimuli is truly essential for reconstruction. In addition, prior work suffers from limited reconstruc- tion quality due to a unified hidden representation that conflates objects and their attributes, often causing object detection errors, e.g.. generating tiger instead of a gray, tiger-striped cat (Section A of the Appendix). This reflects fundamental mismatch with human visual processing, which is object-centric and compositional rather than holistic (Marr) 1980 Bracci & Op de Beeck [022). Overcoming this limitation calls for generative models that explicitly capture the compositional structure of human perception.
To address these issues, we propose PRISM, a model that Projects fMRI sIgnals into Structured text space as an interMediate representation for image reconstruction. To identify the most effective intermediate space, we compare fMRI signals with representations from pre-trained vision, lan- guage, and vision-language models using established metrics (Wang et al., 2020] Murphy et al.] 2024 Keskar et al., 2016). Unlike prior work assuming vision-based representations are essential, our first finding (Section 3.T shows that fMRI signals align more closely with the text space of an LM, motivating the use of solely text bridge for reconstruction. Building this, our sec- ond finding reveals that reconstruction quality improves when the text and the generative model are adapted capture the compositional and relational nature of visual cummull-encompassiinn objects, their attributes, and their relationships. Guided by these insights, we develop two core modules: an object-centric diffusion module that adapts the diffusion model generate images by composing individual objects, and an attribute/relatioossip search module that uses a vision-language model (VLM) to automatically identify object attributes and relationships aligned with neural activity, pro- viding structured guidance for reconstruction. Our contributions are summarized as follows:
Novel Findings: To our knowledge, we are the first to show that accurate visual stimuli reconstruction can be achieved without image-based latent representations, with LM text space effectively bridging brain activity and generative models. Furthermore, we find that adapting this text space and the generative model to capture the compositional and rela- tional nature of visual images further improves reconstruction quality.
Novel Framework: Motivated by our empirical findings, we introduce new IMRI-to- image reconstruction framework that adapts diffusion models for object-centric generation and leverages VLMs to automatically identify brain-aligned object attributes and relation- ships that can optimally guide the reconstruction.
Comprehensive Experiments: Extensive evaluations on real-world fMRI datasets demon- strate that our method achieves up to an 6% reduction in perceptual loss compared to state- of-the-art models, highlighting the effectiveness of our framework.
PRELIMINARY
Notations. In our work, we denote the set of fMRI samples collected during image viewing as X. Each sample is preprocessed ID vector x; E R", capturing neural activity across voxels selected from brain regions (Scotti et al. 2023). The dataset is split into training and test subsets, with superscripts indicating the split. For example, we denote the training set with N samples as: Xamii (XI...... .x.] The corresponding image stimulus for the i-th sample is denoted as: Y, "I # which contains m objects.
Problem Setup. Our goal is to reconstruct the visual images that subjects viewed during fMRI recording. Formally, we seek to learn a reconstruction function F R Hxxx x3 that maps each fMRI sample x; to its corresponding image stimulus Y.
Diffusion Model. Diffusion models (Rombach et al., 2022 Zhang et al. 2023) are class of gener- ative models that synthesize data by learning to reverse multi-step noising process. Starting from Gaussian noise, they iteratively denoise a latent variable over fixed number of timesteps using a de- noising dewoork-typicall a U-Net- conditoeedd on the current timestep f. Thii process gradually produces samples resembling the training distribution. To incorporate external inputs such as text, the denoising network can take conditioning input C, usually text embeddings from pre-trained encoder (Zhang et al.) 2023). Conditioning is implemented via cross-attention mechanisms within the U-Net architecture (Williams et al. 2023). enabling the integration of textual information. In these layers, the latent representations H, xxx at time serve as queries, and C E xdd.
2

---

Published as a conference paper at ICLR 2026
Structured Description Generation
Attribute/ Reeationship
Training images
Training Image
Optimal /Rte./Reltt.
Pre-trained
SSprtial Inyour
Pre-trained VI.
Prompt
Initial AAtrr/Relat. Set
Identifu importont objects, Describe them with (Optimal Attr:/Relat./
Iterative Searchh
Geometric Featuree
Structured Description
Pasitional Kelation do
Training
Preprocess Flatten
Encoder
Fine-tune LM
Text
FMRI-to-text Projection
Predicted Description
Structured Description
Objecti
[Obj: Some curs] [V; follow]
[Obj: A Car] [V; follow] [Ríspatial): elephant's path] [A: white color]
[Rsputial; elephant's path]
ID EMRI
Eneoder
Language Model
[A: and grey]
Object2
[Obj. The elephant] IV:
elephant] [V: stands]
walks] [Kispatial: along the
Space
[R(Spatial): down the roadllA: Grey
road] Grey]
Inference
Object-centric Generation
Comcattn
Prep.
Cruss Att.
Flat.
Encoder
G
Text
Cross Att.
Reconstructed Image
Condition C2
Figure I: Framework Overview: PRISM generates structured text descriptions for each training image using a VLM to iteratively extract brain-aligned object attributes and relationships. These descriptions capture the image's compositional and relational content and serve supervision to train encoder and fine-tune lan- guage model map fMRI signals into the text space. During inference, the model predicts descriptions from (MRI signals, which then guide pre-trained diffusion model for object-centric image reconstruction.
serves as both keys and values (Williams et al. 2023 Yang et al. 2024):
(HH)) We (p(C) WK)
CrossAttention(HH, C) softmax
(CC) Wv.
ddg
where Wq E dRdxd Wx E IR'''dd Wv Te are projection matrices; and o6-) and (p.)) are learned transformations. Further details are available Section of the Appendix.
METHOD
In this section, we present our framework, PRISM, for IMRI-to-imago reconstruction (Figure [. We first show that fMRI signals align most strongly with the text space of LMs, compared to the hidden spaces of vision or vision-language models, under established metrics (Section 3.T This finding motivates annotate each training image with structured text descriptions that are object-centric, compositional, and relational. To generate these descriptions, (Section which learns optimal prompts to guide the VLM in automatically identifying the key attributes and relationships most aligned with both the fMRI signals and images. These structured descriptions are then used to train an encoder and fine-tune the LM, mapping fMRI signals into the LM text space (Section 3.2.2). At inference time (Section 3.3 , the predicted structured descriptions
choice of using pure text the latent space. During training /Section332
introduce attributerrelationssiip search module
guide an adapted diffusion model to generate object-centric images directly from fMRI signals.
3.1 TEXT AS THE LATENT SPACE
We question whether using vision representations as the latent space truly essential for recon- structing visual stimuli. In this section, we investigate the alignment between different model spaces and IMRI signals using various measures.
Measuring the alignment between model spaces and fMRI signals. We examine three representa- tion spaces: (1) the text space of language models, (2) the joint text-image space of vision-language extracted
models, and (3) the latent space of vision models. For (2) and (3), image embeddings directly from the respective models. For (1), we use text embeddings from image captions to repre- sent the stimuli. We extract embeddings by feeding either text images into different models: T5
3

---

Published as a conference paper at ICLR 2026
and LLaMA3 for text embeddings. LDM (Rombach et al. 2022) and ResNet50 (He et al. 2016) for image embeddings, and CLIP for both modalities.
Alignment is assessed using three metrics: Centered Kernel Aligmment (CKA) (Mrphyy et al., 2024). Canonical Correlation Analysis (CCA) (Wang et al.) 2020). and Generalization Gap (Keskar al. 2016). CKA and CCA are widely used to quantity similarity between representation spaces (Kriegeskorte et al.] 2008; Wang et al.] 2020). Generalization Gap reflects learnability by measur- ing the train-fest loss difference when mapping fMRI signals to a target space using an MLP. Good alignment yields higher CKA and CCA values and a lower Generalization Gap.
Let X Concat(x1,,-... ;N)) denote concatenated fMRI samples and K Con ..r....,,))))
the corresponding latent representations. With K((-) as kernel function, the empirical Hilbert-Schmidt Independence Criterion (HSIC) is: H)IC(X, where Lr(.) denotes the trace operator. The CKA is the normalized form: CKA(X, K) =
:N- (X(()) K((K))
/XSICIX,X).ISIICII (J.CK We adopt a Gaussian radial basis function (RBF) kernel for I.C (Alvarez
HSIC(X,K)
P X and v P2 K that maximize their correlation. The first mode captures the dominant shared axis (Wang et al. 2020), with the canonical correlation coefficient: p corr(u, v) = corr(p, T P, K), reflecting the strongest linear alignment between brain activity and the model space.
2022 Cortes et al. 2012). CCA identifies linear projections
FMRI aligns better with the embedding space of language models. Our results (Table [ show that the text space of language models aligns best with fMRI data, outperforming both vision-language and vision-only models across all metrics. Surpris- ingly, vision-language models, despite integrating both modalities, underperform compared to pure language models. We hypothesize that this is be- cause humans focus more on the meaning of an im- age rather than pixel-level details (Naselaris et al.) 2009 Du et al. (022). Unlike prior work (Scotti
Table I: Alignment results between model repre- sentations and fMRI data, evaluated using CKA, Generalization Gap, and CCA. The best result is highlighted in red. denotes higher is better; denotes lower better.
CKA
Generalization Gap
CCA
TS
0.5580
0.1132
0.8344
Llamu3
0,5442
0.2216
0,8022
Clip text
0.5177
0,7599
Clip img
0.3668
0,4860
0.7573
LDM 0.1957
1,2520
0.7215
Resnet50
0.1822
1,9800
0.6746
et al. 2023) Wang et al.) (2024c) Xia et al.] [2024] Lin et al. 2022) that primarily relies on vision representations, our findings motivate using pure text as the latent space.
3.2
TRAINING OF PRISM
In this section, we describe the training process of PRISM, which consists of automatic structured description generation for training images and encoder training.
3.2.1 AUTOMATIC DESCRIPTION GENERATION
We design structured text descriptions as supervision for our framework. To capture the composi- tional and relational nature of human vision, these descriptions should explicitly distinguish between different objects and their relationships. Generating such descriptions with a VLM relies on care- fully crafted prompts that specify the desired attributes and relationships, since many of them not directly reflected in brain activity. To address this issue, we propose a VLM-assisted approach that automatically learns the most relevant attributes and relationships in image based on the training data, ensuring they are both meaningful and brain-aligned.
We first show how structured descriptions can be generated from VLM given learned keyword a, and then present our approach for learning the optimal keyword. Given an image Y, and learned keyword a, we construct prompt P(a) to guide the VLM in describing the most important objects in Y, based on a. Formally, the VLM receives the image and the prompt as input and outputs a structured description D::
D = VLM P(a)).
(I)
The structured description is list of m object-level tuples along with background information:
Dj [(o: d1 loc1), (02 d2 loc2); ..., (Om dm locm), bg:)-
4

---

Published as a conference paper at ICLR 2026
Each 0j is object in the image. dj is its description containing attributes and relationships with other objects conditioned on keyword a, and locy denotes its location (selected from a predefined set). The term bg; represents the background information of image Y, To ensure meaningful generation, we further augment each relation description d, with a structured header encoding its semantic roles, following the PropBank annotation format (Märquez al.) 20088 He et al.) 2017] Ross et al. 2021| Palmer et al.] 2005). A case study on reconstruction with object-level description provided in Section F of the Appendix.
The choice of keyword a strongly influences the attributes and relationships captured in the object descriptions, which in turn affects the quality of mapping fMRI signals to the text space. Ideally, these descriptions should capture the most important information shared between the fMRI signals and the stimulus images. To avoid manually selecting the keyword, we frame its discovery as a prompt optimization problem and introduce our atrribute/relaiionhiip search module.
Concretely, given set of training images yumin (YI...., Yv} and the corresponding fMRI signals Xmmin {xI..... XNJ. we define the following optimization problem to find the optimal in the prompt (P(a)) for the VLM:
max
S
s.t. CKA > 8;
X Concat (XI,...,XN) K Concat (on.att ,k..:
Kg LM P(a))) for I,...,N;
where -(.,)) denotes the similarity score between two images (e.g. negative perceptual loss); Diff is a pre-trained diffusion model that generates images from captions produced by the VLM; Concat in- dicates the concatenation operation across all training samples; and LMENC is pre-trained language model to encode captions generated by the VLM. The constraint enforces that the CKA similarity between the fMRI data X and the caption embeddings K generated using keyword a exceeds threshold 3, ensuring strong alignment between the fMRI and text spaces. The objective ensures that descriptions derived from the optimal keyword support accurate reconstruction.
To optimize the keyword a in Equation (3). we guide the search along semantic links: keywords with similar meanings tend yield comparable reconstructions, so generating new keywords based on the semantic relationships of top-performing candidates helps uncover more effective prompt expressions. In the search, utilize an L.M as a keyword generator and iteratively search for improved keywords in a step-by-step manner. We begin by initializing keywords set A with a collection of frequently-usee attribute and relationship keywords identified in prior works (Johnson) et al., 2015) Lu et al.] 2016] Krishna et al. (017)). We expand A through an e-greedy search strategy: at each search step, the keyword generator proposes new candidate keywords based on either the top-performing keywords in A with probability 1-8, or randomly selected keywords from A with probability s. Only candidates that exceed the similarity threshold are added to A. This balances refinement of effective keywords and exploration of diverse novel keywords. See Section [ in the Appendix for the detailed algorithm and search results.
3.2.2 ENCODER TRAINING
We design an encoder map fMRI signals into the latent space of the language model, using structured and object-centric descriptions as supervision. Specifically, each object's information is independently encoded using an MLP. The resulting representations are concatenated and passed to the language model to generate estimated structured descriptions Dg. which can be expressed as:
f, MLP,(xx), j 1, ,
D = LMMPP (Concat(f1.... ..... f.,))),
The language model is fine-tuned using a loss over all m object descriptions (Chang et al. 2024 Gunel et al. 2020
m T
LLM = Σ log p(ye ()),
(5)
j=1f=1
where yr denotes the t' token in the structured description. This training strategy enables fine- grained alignment between fMRI signals and structured textual descriptions. We first train the ML.Ps
5

---

Published as a conference paper at ICLR 2026
independently for fixed number of epochs, then jointly fine-tune the language model and MLPs to maximize overall reconstruction performance.
3.3 PRISM INFERENCE: OBJECT- CENTRIC IMAGE GENERATION
The inference process of PRISM has two steps: (I) generate structured descriptions De from fMRI signals using the trained encoder and language model, and (2) reconstruct the image by composing objects conditioned on these descriptions with pre-trained diffusion model: Y, = Diff( D;).
To reflect the brain's 's compositional understanding of visual scenes, during inference, we adapt pre- trained diffusion model to perform compositional image generation, inspired by (Yang et al. (2024). Specifically, given an image Y; and its predicted structured description Dē (from Equation (4 we
extract a set of predicted objects {or and background description bg;- These are combined into global context prompt po and embedded into a conditioning matrix Co. Similarly, each object description d, is embedded into a corresponding conditioning matrix Cj. These conditioning matrices guide the denoising process via cross-attention (Section 2 from time t to O. At each step, the hidden representation of object j (or the global context when j 0) at time - 1 is computed as: H Cross (He, Cý), where H; is the hidden representation of the full image at time We then resize and concatenate the object representations according to their predicted locations locy:
(6) where (-). denotes the resizing and spatially concatenation operation. Thus, the image gener- ation model encodes each object independently from its description and then spatially concatenates their hidden representations according to the predicted locations.
cat
1 )
To ensure smooth region boundaries and seamless fusion between objects and background, we com- pute weighted sum of the global context latent and the object latents:
H 3 H + (1 11-3) B) H
where 3 is a hyperparameter that controls the blending ratio. This process is repeated across denois- ing steps, enabling structured, object-aware generation aligned with the brain's 's visual understanding.
4 EXPERIMENTS
In this section, we conduct extensive experiments evaluate PRISM, guided by the following questions: (RQ1) How well does our framework PRISM perform on the image reconstruction task? (RQ2) How do different choices of latent space influence the reconstruction quality? (RQ3) What is the contribution of each component in our framework to the overall reconstruction performance?
4.1 EXPERIMENTAL. SETUP
We conduct experiments on three datasets: NSD (Allen et al. 2022), BOLD5000 (Chang et al, 2019), and GOD (Horikawa & Kamitani 2017). Detailed descriptions of the datasets are provided in Section C of the Appendix. Each method is evaluated by comparing the reconstructed images to the ground truth using five metries: PixCorr and SSIM for pixel- and structure-level similarity (Wang et al. 2004). LPIPS (Zhang et al., (2018) for human perceptual similarity, and CLIP and Inception V3 two-way identification (Scotti et al. 2024) for semantic consistency based on pretrained model representations. We compare our method against the following baselines: Takagi & Nishimoto (Takaii & Nishimoto 2023) (Takagi for short), Mindvis (Chen et al. 2023). Mindeye (Scotti al., 2023). MindBridge (Wang et al.) (024b). NeuralDiffuser (Li et al. 2025). and Mindeye2 (Scotti al. 2024). To ensure fair comparison, we use the same generative model, Stable Diffusion 2.1 (Pernias et al. 2023 Rombach et al., (2022), for all methods. We additionally present results for PRISM and Mindeye2 (ranked second-best -bst) with the newer SDXL backbone (Podell et al. 2023). We incorporate negative prompt in our model-a textual constraint that guides the diffusion model to avoid generating undesired visual artifacts, such as distorted object shapes or background clutter. For fairness, apply the same negative prompt when evaluating all basclines, which leads to performance gains in some cases. More details about the baselines and training are provided in Sections Dad Eof the Appendix.

---

Published as a conference paper at ICLR 2026
Table 2: Comparison of framework with state-of-the-art methods three datasets. All methods Stable Diffusion 2.1 as the backhone unless otherwise specified (+SDXI.). Results are reported using PixCorr, SSIM, LPIPS, CLIP and Inception V3 metrics. The best result using the same backhone in cach column is highlighted in red. indicates higher is better and indicates lower is better.
NSD
PixCorr f
SSIM f
LPIPS
CLIP
Inception V3
PRISM
0.3404-40.05
0.4640+0.02
0,5963+0.0p
0,9467 =0.03
0.951640.00
Takagi
0.2100
0.3880
0.7665 10.04
0,881 1 10.06
0.908610.07
Míndvis
0.2736+0.00
0.3868
0.6789
0,9000 10.05
0.9135 =0.05
Mindeyel
0.3114
0.3868
0.6501 20.00
0.9121 10.04
0.9198
MindBridge
0.1802
0.2823 00.02
0.6977 +0.03
0.9427 0.002
0.9242
NeuralDiffuser
0.301 1 10005
0.3348 10.03
04 0,6522
0.9409 10.01
0.9487
Mindeye2
0.316010.04
1..444.11.02
0.6338 10004
0.9201 10003
0.9338 1.008
PRISM+SDXL
0.3645 40.02
0.4983
0.5563 40.02
0,9600
0.976540.00
Mde2+SDXL
0,3471
0,4425+4.04
0.6002
0,959920:02
0.9602+0.01
BOLD5000
PRISM
0,2315
0.5341 1:000
0.6198 1:0.00
0.7720
0.6601
Takagi
0.1815
0.44181
0.7558
0.6990
0.5667
Mindvis
0,2122 =0.05
0.4944 =0.04
0.6463 20.05
0.7720
0.5701 =0.08
Mindeyel
0.1942+:0.01
0.4838
0,6913
0.7288 20.03
0.6222 00.07
MindBridge
0.1522++0.04
0.3005
0.6535
#:0.03
0.7431 #0.01
0.6200
NeuralDiffuser
0.2036:10.04
0,4005 2.002
0.6899 =:0
0.7609 20.01
0.6522+1000
Mindeye2
6.2265.10.02
0.516410.02
0.641620.00
0.760022.04
0.6428 10.03
PRISM+SDXI
0.2442
2.002
0.5600
20.03
0,5909
2.0.4
0.7881 1.004
0.6881 10.02
SDX2+
0,2310
=0.04
0.5185
=0.02
0.6186
20.02
0.750320.04
0.6556+0.02
GOD
PRISM
0.2571 10.01
0.5200
=0.02
0.6213
10.01
0.8567
10.05
0.8428 10.06
Takagi
0,2322 10.05
0.4944
=0.04
0.6463
10.05
0.7232 10.01
0.7556 10002
Mindvis
0.1921 +0.02
0.430440.03
0.697
0.7162+2.08
0.6119 #0.03
Mindeyel
0,2286 10.03
0.4766
=0.02
0.6807 20.05
0.8093 10.01
0.8002 10004
MindBridge
0.189844..4
0..027
0.6960
0.8219 #0.01
0.7960 #0.01
NeuralDiffuser
0,2006 02
0.4253.40.03
0.6836g0.00
0,8501 20.01
0.837220.00
Mindeye2
0.2442 #0.03
0,4952
0.6586
#0.02
0.8322
0.8280+0.04
PRISM-SDXI.
0.2669 20.01
0,5537
20.01
0.5989
10.03
0.8727 10003
0.882010.04
Mde2+SDXL
0.2500 =0.04
0.5511
=0.04
0.6224
20.01
0.8678
0.855640.01
4.2 EFFECTIVENESS OF PRISM
We evaluate our IMRI-to-image reconstruction framework test data and compare it with state-of- the-art methods.
Table 2 summarizes the results, with all metrics and standard deviations averaged across subjects over five runs. Visualiza- tions of reconstructed examples from the test are shown in Figure As shown in Table PRISM outperforms state-of-the. art methods across all datasets and met- rics, with up to 6% improvement in LPIPS, indicating higher perceptual similarity to the original images. Unlike baselines such as Mindeye2 (Scotti et al.) (024) and Mindeyel (Scotti et al.) 2023). which often ignore key objects, PRISM successfully reconstructs all objects, yielding notable gains across all metrics. These results demonstrate the effectiveness of our framework in translating brain activity into accurate, perceptually aligned image reconstructions.
Table Image QA results on reconstructed NSD images. Mdvs, Mdel and Mde2 refer to Mindvis. Mindeyel and Mindeye2, respectively. Results are reported as accuracy, with the best highlighted in red.
PRISM Takagi
Mdvs
Mdel
Mde2
Acc.t 0.6054
0.4011
0.5037
0.5516
0.5765
We further evaluate our method on a question answering (QA) task using reconstructed test images from the NSD dataset. For each image, we retrieve corresponding question answerr pair from the COCO dataset (Lin et al.) 2014) and use Qwen2.5 (Bai et al. 2025) to answer the question based on the generated image. QA accuracy is reported in Table Our method achieves accuracy

---

Published as a conference paper at ICLR 2026
Ground Truth
PRISM
Mindeyez
Mindeyes
Mindvis
Takagi
Figure 2: Reconstructed images from different methods. The first column shows the original viewed images. The rest of the columns show the reconstructed images from different methods.
of 60.54%, significantly outperforming state-of-the-art methods. This demonstrates that our recon- structions are not only visually faithful but also semantically meaningful.
4.3 ABLATION STUDY
In this subsection, we present ablation studies to justify our choice of text as the latent space and to evaluate the effectiveness of the object-centric diffusion and attribute/relaiionhiip search modules. Experiments are conducted on the NSD dataset, though the trend generalizes to other datasets.
We first compare reconstruction per- formance across three latent spaces: (1) language model embeddings (ours), (2) CLIP text embeddings (CxIP-Text), and (3) the image latent space of diffusion model (LDM). As shown in Table 4 aligning fMRI signals to the language model text space consistently outperforms the other two spaces across all metrics. This demonstrates that textual representations alone can capture multiple levels of visual information, making text space a more brain-ali train-aligned and effective intermediate representation for iMRI-to-image reconstruction. Results for the remaining two datasets are provided Table 7 of the Appendix.
Table 4:
Reconstruction performance across three latent spaces. The best result in each column is highlighted red. indicates higher is better and indicates lower is better.
PiaCorr
Inception
PRISM Clip teNt
0.34D4ana:
0.4610.a.0 0.3725
0.5963.002
29.67
0.516-pm
0.3208
0.66112001
0.9197
0
LDM
0,209040m:
0.3727
0,7502a0ot
0,8602
0.8925
Next, we evaluate the effectiveness of the two proposed modules. Re- sults are in Table 5 To evalu- the Object-centric Diffusion mod- ule, we compare against a variant (w/o ObjC.) that replaces object- level cross-attention with standard U- Net cross-attention. To assess the at- tribute/relationship search module, test two variants that skip the search process and rely only on the initial keyword set: w/o AttOpt.+Bst, which fixes the prompt to the highest-scoring (best) keyword, and w/o AttOpt.+Wst which fixes it to the lowest-scoring (worst) keyword.
Table 5: Effectiveness of the object-centric diffusion module and attribute/relaaioonhhpp search module on NSD data. The best result highlighted in red.
PixCorr 1
SSIM 1
LPIPS 1
PRISM
0.340440.o
0.4640.0.02
0.5963
w/o ObjC.
0.3291 1006
0.4299 10.6
0.6111
10.05
w/o AuOpt+Bad
0.3311
0.4421
0.6005
Attptt.
#2.05
0.4167
0.6398
Overall, removing or replacing the two modules consistently degrades performance across all met- rics. Specifically, eliminating object cross-attention leads to notable declines that cannot be recov-
8

---

Published as a conference paper at ICLR 2026
ered through prompt optimization, highlighting its essential role in reconstructing perceptually ac- curate images. Likewise, bypassing prompt optimization and using the best or worst initial attribute also reduces performance, indicating that the initial attributes alone insufficient and underscoring the importance of prompt optimization in our model. The ablation study on the number of objects in our framework is shown in Section Hf the Appendix.
4.4 KEYWORD SEARCH
Case Study. To better understand the keywords selected by our attributefrelationsiin search module, Tablee]jreseetss the top-scoring keywords across different rounds of the s-greedy search. The results show that, despite extensive exploration, the top-scoring keywords consistently converge toward spatially oriented relationships such as Spat i a Layout and Relat i Position. This suggests that: (1) descriptions emphasizing spatial information are most effective for guiding the diffusion model to accurately reconstruct images, as indicated by their highest LPIPS scores; and (2) these weywords also align well with fMRI data, as their CKA scores are no lower than those of the initial keywords, in accordance with the search constraints. This result is consistent with prior neuro-scientific findings showing that neural representations in the brain are sensitive to spatial arrangements and relative positions of objects (Zopf et al. 20188] Graumann et al., (022). Therefore, we use Spatial Layout as the optimal keyword a to generate structured descriptions for model training. Additional implementation details of the attribute/relationshipp search module can be found in Section[]of the Appendix. To further demonstrate its robustness, we perform an experiment where all spatial-related keywords are deliberately excluded from the initial keyword set. The results, presented in Section of the Appendix, show that the module is still able to identify the relevant spatial keywords, demonstrating the robustness and effectiveness of our search process.
Neuroscientific Interpretation of the Optimal Keyword. To explore why the optimal keyword Spatial Layout exhibits stronger alignment with fMRI signals, we conduct gradient-based interpretability analysis to examine the neural correlations of different keyword types. Specifically, during inference on the test set, we identify predicted words associated with spatial relationships (e.g., directional terms) and functional attributes (e.g., actions or behaviors of objects), and compute the gradients of these tokens with respect to the input fMRI signals. Ideally, larger gradients indicate that specific voxel values within the fMRI contribute more strongly to predicting the given keyword, and are thus more relevant to that semantic category. By averaging these gradients across samples, we identify the ROIs that contribute most to each keyword category. For spatial relationships, we observe the strongest activation in the ROI 119 (HCP mmpl atlas) (Glasser et al. 2016). which be- longs to Presubiculum (PreS), region linked to spatial memory (Dalton & Maguire (217]: Boecker eet al. 2024). In contrast, for functional attributes, the Ventromedial Visual Area (VMVI, ROI 153 of HCP mmpl atlas) shows the highest activation. Additionally, intensity within each ROI and find that PreS exhibited a higher mean activation (0.0080) compared to VMVI (0.0028). These findings suggest that the fMRI contains stronger activation in spatially relevant regions such as PreS, which may explain the higher alignment observed for spatial key- words. Nonetheless, we acknowledge that further validation from the neuroscience community is needed.
compute the mean voxel
5 RELATED WORK
FMRI-Image Reconstruction. Early approaches leveraged linear models to decode fMRI sig- nals into visual features (Kay et al., 2008 Takagi & Nishimoto 2023). Later work explores deep generative models: approaches such as (Lin et al. 2022] Ozcelik et al. 2022 Goodfellow et al.) 2020) map fMRI signals into the latent space of GANs for image reconstruction. while BraVL (Du let al.
2023) employs a multimodal VAE to jointly model relationships between brain activity and visual linguistic features for neural decoding. With advances in vision- models (Radford ctt al. 2 Liang et al., 2024), more recent approaches map fMRI signals into CLIP's image- embedding space (Scotti et al.] 2024 2023) and then use diffusion models for image reconstruc- tion (Rombach et al.] 2022; Xu et al. 2023, Podell et al. 2023). Building on this diffusion-basst direction, MindDiffuser (Lu a 2023) introduces two-stage framework that first decodes mantic information and then aligns structural information with CLIP visual features decoded from fMRI. Unlike prior work that directly maps fMRI signals to joint text-image spaces (Wang al.,
9

---

Published as a conference paper at ICLR 2026
Table 6: Top-5 keywords scored by 1 LPIPS hefore searching and after 10, 20, 30 search steps. The
top-5 results remain unchanged after 30 search rounds. The search results indicate a clear preference for keywords related to spatial and positional relations, with most of the top-perfornig keywords in the final results containing the term spatial".
Rank
Initial
Round 10
Round 20
Round 30+
#1
Spatial Con fonfuraatiion
Spatial Arrangemen:
Spatial Organization ion Spatial Structure
Spatial Layout
#2
Foaitional Relation
Spacial Contiguzation
Spacial Fatterns
#3
Lecazion Relation
Spatial Inseraction
Spatial Azrangenent
Spatial Osqanizacion
#4
Descriptive Attribute
Positional Helation
Spatial Configuration
Helative Position
#5
Inclusion Dependency
Feacure Rol
Spat fal Interaction
Spactal Relat tonationahips
2024c; Quan et al. 2024), we compare multiple representation spaces and find that text embed- dings from language models (Raffel al.] 2020) exhibit the strongest alignment with fMRI sig- nals. This insight motivates our approach of reconstructing images via the embedding space of language models. Cross-subject decoding has also gained attention. MindTunee (Gong et al.) 2025) introduces fingal-fingerprint modeling using LoRA to improve cross- alignment and mantic correction. Psychometry (Quan et al. 2024) proposes an omni cexture-of-Expere architec- ture that captures both inter-subject shared structure and subject-specific variability, combined with retrieval-enhanced inference for improved reconstruction. Wills Aligner (Bao et al.) (2025) further advances multi-subject collaborative decoding by combining anatomical alignment with subject- guided Cestur-of-Brrin-Exxeee adapters. Beyond static images, recent work has extended brain decoding dynamic visual stimuli. NeuroClips (Gong et al., 2024) reconstructs such stimuli from IMRI by combining decoded keyframes with low-level perceptual flows to improve temporal con- sistency. BrainNEDS (Yeung al., 2025) utilizes video diffusion models to disentangle static and dynamic components, offering insights into which visual features are prioritized by the brain.
Diffusion models. Diffusion models have become foundational in generative tasks like image cre- ation and editing (Gal et al.] 2022) Song et al. 2020)), as well as text-to-image synthesis (Ruiz et al.) 2023 To enhance control over generated content, ControlNet (Zhang et al. (023) introduces high- level image features for controlling and GLIGEN (Li et al.
et al.) 2025) incorporates position-awan aware adapters for spatial grounded generation. Mcanwhile, there are also training- free methods that adjust latent or attention maps during inference to guide outputs without additional training (Chen et al. 2024] Yang et al. 2024). In our work, we guide the diffusion process by modifying gross-attentio layers during inference to integrate object-level descriptions derived from fMRI data for image reconstruction.
Prompt Optimization. Prompt optimization aims to discover effective textual prompts for LLMs without model fine-tuning. Gradient-based methods (Shin et al. 2020 Shi et al. 2022 Wen et al., 2023) update prompts using gradients differentiable embeddings. Gradient- free approaches treat
LLMs as black boxes, using heuristic search (Prasad et al. 2022 Pryzant et al. 2022), reinforcement
learning (Deng et al., 2022 Zhang et al. (022), or evolutionary strategies (Zhou al.) 2022] Yang
et al. 2023) Guo et al. (025).. We designed our gradient-free prompt optimization based on beam
search to optimize attribute keywords for black-box vision- Manguage models.
CONCLUSION
In this work, we address the challenge of reconstructing visual stimuli from fMRI signals. Our analysis reveals that fMRI signals align more closely with the text space of language models than with vision-based or joint text-image representations. identifying text brain- brain-aligned intermedi- ate space. Building on this insight, we show that explicitly modeling the compositional structure of visual perception- capttring objects along with their attributes and relationships-furr further improves reconstruction quality. Guided by these findings, we develop PRISM, a framework that maps fMRI signals into a structured text space and incorporates two specialized modules: an object-centric dif- fusion module that generates images by composing individual objects, and an altribute/relatiooshipp search module that automatically discovers attributes and relationships aligned with neural activity. Experiments on real-world fMRI datasets demonstrate that PRISM reduces perceptual loss by up to 6% compared to prior methods, underscoring the power of structured text as a bridge between brain activity and image generation.
10

---

Published as a conference paper at ICLR 2026
7 ETHICS STATEMENT
Our work does not involve human or animal subjects, personally identifiable data, or sensitive in- formation. The datasets used are publicly available, and we follow their respective licenses. The methods and findings presented do not pose foreseeable risks of misuse, discrimination, or harm. We therefore believe our work raises no specific ethical concerns under the ICLR Code of Ethics.
REPRODUCIBILITY STATEMENT
Section 3 details the proposed framework and its design. Section describes the datasets, base- line methods, and evaluation protocols used for comparison. Additional implementation details, including training procedures and hyperparameter settings, are provided in the Appendix. Upon acceptance of this paper, we will release our code on GitHub.
REFERENCES
Emily Allen, Ghislain St-Yves, Yihan Wu, Jesse L Breedlove, Jacob Prince, Logan Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, et al. A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence. Nature neuroscience, 25(1): 16-126, 2022.
Sergio A Alvarez. Gaussian rbf centered kernel alignment (cka) in the large-bandwidth limit. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(5):658776553, 2022.
Shuai Bai, Keqin Chen, Xuejing Liu, Jialin Wang, Wenbin Ge, Sibo Song, Kai Dang, Peng Wang, Shijie Wang, Jun Tang, al. Qwen2. 5-vl technical report. arXiv preprint arXiv:2502.1392 13923, 2025.
Guangyin Bao, Qi Zhang, Zixuan Gong, Jialei Zhou, Wei Fan, Kun Yi, Usman Naseem, Liang Hu, and Duoqian Miao. Wills aligner: Multi-subject collaborative brain visual decoding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 14194-14202. 2025.
Henning Boecker, Marcel Daamen, Lukas Kunz, Melanie GeiB, Moritz Müller, Thomas Neuss, Leonie Henschel, Rüdiger Stirnberg, Neeraj Upadhyay, Lukas Scheef, et al. Hippocampal subfield plasticity associated with improved spatial memory. Communications biology, 7(1):211, 2024.
Stefania Bracci and Hans P Op de Beeck. Understanding human object vision: picture is worth 2023.
thousand representations. Annual review of psychology,
Nadine Chang, John A Pyles, Austin Marcus, Abhinav Gupta, Michael Tarr, and Elissa M Aminoff. Bold5000, public fmri dataset while viewing 5000 visual images. Scientific data, 6(1):49, 2019.
Yupeng Chang, Xu Wang, Jindong Wang, Yuan Wu, Linyi Yang, Kaijie Zhu, Hao Chen, Xiaoyuan Yi, Cunxiang Wang, Yidong Wang, et al. A survey on evaluation of large language models. ACM transactions on intelligent systems and technology. 15(3):1-455 2024.
Minghao Chen, Iro Laina, and Andrea Vedaldi. Training-free layout control with cross-attention guidance. In Proceedings of the IEEE/CVF winter conference on applications of computer vision, pp. 5343-5353 2024.
Zijiao Chen, Jiaxin Qing, Tiange Xiang, Wan Lin Yue, and Juan Helen Zhou. Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22710- 22720, 2023.
Corinna Cortes, Mehryar Mohri, and Afshin Rostamizadeh. Algorithms for learning kerness based on centered alignment. The Journal of Machine Learning Research, 13(1):795-828, 2012.
Marshall A Dalton and Eleanor A Maguire. The pre/parasubiculum: a hippocampal hub for scene- based cognition? Current Opinion in Behavioral Sciences, 17:34 40, 2017.

---

Published as a conference paper at ICLR 2026
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A acale-scale hi- erarchical image database. In 2009 JEEE conference on computer vision and pattern recognition, pp. 248-255, Icee, 2009.
Mingkai Deng, Jianyu Wang. Cheng- Beeng-Ping Hsich, Yihan Wang, Han Guo, Tianmin Shu, Meng Song, Eric P Xing. and Zhiting Hu. Rlprompt: Optimizing discrete text prompts with reinforcement learning. arXiv preprint arXiv:2205. 12548, 2022.
Bing Du, Xiaomu Cheng, Yiping Duan, and Huansheng Ning. fmri brain decoding and its applica- tions in brain-connnn crmputc interface: survey. Brain Sciences, 12(2))228, 2022.
Changde Du, Kaicheng Fu, Jinpeng Li, and Huiguang He. Decoding visual neural representations by multimodal learning of vruin-visual sesuull-inguistic features. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(9):107600110 10777, 2023.
Ali Farhadi, Ian Endres, Derek Hoiem, and David Forsyth. Describing objects by their attributes. In 2009 IEEE conference on computer vision and pattern recognition, pp. 1778-1784 IEEE, 2009.
Rinon Gal, Yuval Alaluf, Yuval Atzmon, Or Patashnik, Amit H Bermano, Gal Chechik, and Daniel Cohen-Or. An image is worth word: Personalizing text-to-image generation using textual inversion. arXiv preprint wrXiv:2208.06618 2022.
Matthew F Glasser, Timothy S Coalson, Emma C Robinson, Carl D Hacker, John Harwell, Essa Yacoub, Kamil Ugurbil, Jesper Andersson, Christian F Beckmann, Mark Jenkinson, et al. A multi-modal parcellation of human cerebral cortex. Nature, 536(7615) 2016.
Zixuan Gong, Guangyin Bao, Qi Zhang, Zhongwei Wan, Duoqian Miao, Shoujin Wang. Lei Zhu, Changwei Wang, Rongtao Xu, Liang Hu, al. Neuroclips: Towards high-fidelity and smooth fmri-to-video reconstruction. Advances in Neural Information Processing Systems, 37:51655- 51683, 2024.
Zixuan Gong, Qi Zhang, Guangyin Bao, Lei Zhu, Rongtao Xu, Ke Liu, Liang Hu, and Duoqian Miao. Mindtuner: Cross-subject visual decoding with visual fingerprint and semantic correction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 14247-14255, 2025.
lan Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(11):13,-144 2020.
Monika Graumann, Caterina Ciuffi, Kshitij Dwivedi, Gemma Roig, and Radoslaw M Cichy. The spatiotemporal neural dynamics of object location representations the human brain. Nature human behaviour, 6(6):796-811,2) 2022.
Beliz Gunel, Jingfei Du, Alexis Conneau, and Ves Stoyanov. Supervised contrastive learning for pre-trained language model fine-tuning. arXiv preprint arXiv:2011.01403, 2020.
Qingyan Guo, Rui Wang, Junliang Guo, Bei Li, Kaitao Song, Xu Tan, Guoqing Liu, Jiang Bian, and Yujiu Yang. Evoprompt: Connecting Iims with evolutionary algorithms yields powerful prompt optimizers, 2025. URL/httss://arxiit http://aawwiar....aa//a/a abs 2309. 08532
Mandar Haldekar, Ashwinkumar Ganesan, and Tim Oates. Identifying spatial relations in images using convolutional neural networks. In 2017 international joint conference on neural networks (ijenn), pp. 3593-3600. IEEE, 2017.
Stefan Haufe, Frank Meinecke, Kai Görgen, Sven Dähne, -yllan Haynes, Benjamin Blankertz, and Felix Bießmann. On the interpretation of weight vectors of linear models in multivariate neuroimaging. Neuroimage, 87:96-1 10, 2014.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
12

---

Published as a conference paper at ICLR 2026
Luheng He, Kenton Lee, Mike Lewis, and Luke Zettlemoyer. Deep semantic role labeling: What works and what's ' next. In Proceedings of the 55th Anmual Meeting of the Association for Com- putational Linguistics (Volume I: Long Papers), pp. 473-483, 2017.
Tomoyasu Horikawa and Yukiyasu Kamitani. Generic decoding of seen and imagined objects using hierarchical visual features. Nature communications, 8(1): ((1):15037, 2017.
Justin Johnson, Ranjay Krishna, Michael Stark, Li-Jia Li, David Shamma, Michael Bernstein, and Li Fei-Fei. Image retrieval using scene graphs. In Proceedings of the IEEE conference on com- puler vision and paltern recognition, pp. 3668-3678, 2015.
Kendrick N Kay, Thomas Naselaris, Ryan Prenger, and Jack Gallant. Identifying natural images from human brain activity. Nature, 452(7185):352-335 2008.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Pe- ter Tang. On large-batch training for deep learning: Generalization gap and sharp minima. arXiv preprint arXiv: 04836, 2016.
Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandettini. Representational similarity analysis- connecting the branches of systems neuroscience. Frontiers in systems neuroscience, 2:249, 2008
Ranjay Krishna, Yuke Zhu, Oliver Groth, Justin Johnson, Kenji Hata, Joshua Kravitz, Stephanie Chen, Yannis Kalantidis, Li-Jia Li, David A Shamma, et al. Visual genome: Connecting lan- guage and vision using crowdsourced dense image annotations. International journal computer vision, 123:32-73, 2017.
Haoyu Li, Hao Wu, and Badong Chen. Neuraldiffuser: Neuroscience-inspired diffusion guidance for fmri visual reconstruction. IEEE Transactions on Image Processing, 2025.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-ine eext-to-image generation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22511 122221, 2023.
Zijing Liang, Yanjie Xu, Yifan Hong, Penghui Shang, Qi Wang, Qiang Fu, and Ke Liu. A survey of multimodel large language models. In Proceedings of the 3rd International Conference on Computer, Artificial Intelligence and Control Engineering, pp. 405-409. 2024.
Sikun Lin, Thomas Sprague, and Ambuj K Singh. Mind reader: Reconstructing complex images from brain activities. Advances in Neural Information Processing Systems, 35:29624-29636 2022.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pictro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, pro- ceedings, part 13, pp. 740-755. Springer, 2014.
Yujian Liu, Yang Zhang, Tommi Jaakkola, and Shiyu Chang. Correcting diffusion generation through resampling. Proceedings of the IEEE/CVF Conference on Computer Vision and Pat- tern Recognition, pp. 8713 8723, 2024.
Cewu Lu, Ranjay Krishna, Michael Bemstein, and Li Fei-Fei. Visual relationship detection with language priors. In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part 14, pp. 852-869. Springer, 2016.
Yizhuo Lu, Changde Du, Qiongyi Zhou, Dianpeng Wang, and Huiguang He. Minddiffuser: Con- trolled image reconstruction from human brain activity with semantic and structural diffusion. In Proceedings of the s1 ACM international conference on multimedia, pp. 5899-5908, 2023.
Andrew F Luo, Margaret M Henderson, Michael J Tarr, and Leila Wehbe. Brainscuba: Fine-grained natural language captions of visual cortex selectivity. arXiv preprint arXiv:2110.04420, 2023.
Weijian Mai, Jian Zhang, Pengfei Fang, and Zhijun Zhang. Grain-conditional multimodal synthesis: A survey and taxonomy. IEEE Transactions on Artificial Intelligence, 2024.
13

---

Published as a conference paper at ICLR 2026
Lluis Märquez, Xavier Carreras, Kenneth C Litkowski, and Suzanne Stevenson. Semantic role labeling: an introduction to the special issue, 2008.
David Marr. Visual information processing: The structure and creation of visual representations. Philosophical Transactions of the Royal Society of London. B, Biological Sciences, 290 193()) 199-218, 1980.
Alex Murphy, Joel Zylberberg, and Alona Fyshe. Correcting biased centered kernel alignment measures in biological and artificial neural networks. arXiv preprint arXiv:2405.01012 2024.
Thomas Naselaris, Ryan Prenger, Kendrick N Kay, Michael Oliver, and Jack L Gallant. Bayesian reconstruction of natural images from human brain activity. Neuron, 63(6))902-99) 915, 2009.
Thomas Naselaris, Kendrick N Kay, Shinji Nishimoto, and Jack Gallant. Encoding and decoding in fmri. Neuroimage, 56(2):400-410, 10, 2011.
Furkan Ozcelik, Bhavin Choksi, Milad Mozafari, Leila Reddy, and Rufin VanRullen. Reconstruction of perceived images from fmri patterns and semantic brain exploration using instance-conditioned gans. In 2022 international joint conference on neural networks (IJCN)), pp. 1-8. IEEE, 2022.
Martha Palmer, Daniel Gildea, and Paul Kingsbury. The proposition bank: An annotated corpus of semantic roles. Compulational linguistics,, 31(1):71-11) 2005.
Pablo Pernias, Dominic Rampas, Mats L Richter, Christopher J Pal, and Marc Aubreville. Würstchen: An efficient architecture for large-scale text-to-image diffusion models. arXiv preprint arXiv:2306.0000 006377000067,... 2023.
Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhom, Jonas Müllrr, Joc Penna, and Robin Rombach. Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint 20070:2307.01952, 2023.
Archiki Prasad, Peter Hase, Xiang Zhou, and Mohit Bansal. Grips: Gradient-free, edit-based in- struction search for prompting large language models. arXiv preprint arXiv:2203.07281, 2022.
Reid Pryzant, Dan Iter, Jerry Li, Yin Tat Lee, Chenguang Zhu, and Michael Zeng. Automatic prompt optimization with" gradient descent" and beam search. arXiv preprint arXiv:2305.03495. 2023.
Weikang Qiu, Zheng Huang, Haoyu Hu, Aosong Feng, Yujun Yan, and Rex Ying. Mindllm: A subject-agnostic and versatile model for fmri-to-text decoding. In Forty-second International Conference on Machine Learning, 2025.
Ruijie Quan, Wenguan Wang, Zhibo Tian, Fan Ma, and Yi Yang. Psychometry: An omnifit model for image reconstruction from human brain activity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 233-243, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-877 -8733 PmLR, 2021.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140): 2020.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Alexis Ross, Tongshuang Wu, Hao Peng, Matthew E Peters, and Matt Gardner. Tailor: Generating and perturbing text with semantic controls. arXiv preprint arXiv:2 2021.
Nataniel Ruiz, Yuanzhen Li, Varun Jampani, Yael Pritch, Michael Rubinstein, and Kfir Aberman. Dreambooth: Fine tuning text-to-image diffusion models for subject-driven generation. In Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 22500- 22510, 2023.
14

---

Published as a conference paper at ICLR 2026
Paul Scotti, Atmadeep Banerjee, Jimmie Goode, Stepan Shabalin, Alex Nguyen, Aidan Dempster, Nathalie Verlinde, Elad Yundler, David Weisberg, Kenneth Norman, et al. Reconstructing the mind's eye: fmri-to-image with contrastive learning and diffusion priors. Advances in Neural Information Processing Systems, 36:24705-24728, 2023.
Paul Scotti, Mihir Tripathy, Cesar Kadir Torrico Villanueva, Reese Kneeland, Tong Chen, Ashutosh Narang, Charan Santhirasegaran, Jonathan Xu, Thomas Naselaris, Kenneth A Norman, et al. Mindeye2: Shared-subjes models enable fmri-to-image with hour of data, arXiv preprint arXiv:2403. 1207, 2024.
Weijia Shi, Xiaochuang Han, Hila Gonen, Ari Holtzman, Yulia Tsvetkov, and Luke Zettlemoyer. Toward human readable prompt tuning: Kubrick's s the shining is good movie, and good prompt too? arXiv preprint arXiv:2212. 2022.
Taylor Shin, Yasaman Razeghi, Robert Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:2010. 15980, 2020.
Ranganatha Sitaram, Nikolaus Weiskopf, Andrea Caria, Ralf Veit, Michael Erb, and Niels Bir- baumer. fmri brain-computer interfaces. IEEE Signal processing magazine, 25(1):95-106, 2008.
Jiaming Song. Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02500 2020.
Jingyuan Sun. Mingxiao Li, Zijiao Chen, Yunhao Zhang, Shaonan Wang, and Farnc-Francnne Moens. Contrast, attend and diffuse to decode high-resolution images from brain activities. Ad- vances Neural Information Processing Systems, 36: 2332-12348, 2023.
Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. (4453-14463 2023.
Hao- Mlo-TTng Wang, Jonathan Smallwood, Janaina Mourao-Miranda, Cedric Huchuan Xia, Theodore D Satterthwaite, Danielle S Bassett, and Danilo Bzdok. Finding the needle in a high-dimensional haystack: Canonical correlation analysis for neuroscientists. Neurolmage, 216:1 16745, 2020.
Ling Wang, Chen Wu, and Lin Wang. Braindreamer: Reasoning-coherent and controllable im- age generation from eeg brain signals via language guidance. arXiv preprint arXiv:2409.14000 2024a.
Shizun Wang, Songhua Liu, Zhenxiong Tan, and Xinchao Wang. Mindbridge: A cross-subject brain decoding framework. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pallern Recognition, pp. 11333-11342, 2024b.
Su Wang, Chitwan Saharia, Ceslee Montgomery, Jordi font-Tuset, Shai Noy, Stefano Pellegrini, Yasumasa Onoe, Sarah Laszlo, David J Fleet, Radu Soricut, et al. Imagen editor and editbench: Advancing and evaluating text-guided image inpainting. In Proceedings of the IEEF/CVF con- ference on computer vision and pattern recognition, pp. 18359-18369, 2023.
Yanchen Wang, Adam Turnbull, Tiange Xiang, Yunlong Xu, Sa Zhou, Adnan Masoud, Shekoofeh Azizi, Feng Vankee Lin, and Ehsan Adeli. Decoding visual experience and mapping semantics through whole-brain analysis using fmri foundation models. arXiv preprint arX1v:2411.07121, 2024c.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE transactions image processing, 13(4):600- 612, 2004.
Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Hard prompts made easy: Gradient-based discrete optimization for prompt tuning and discovery. Advances in Neural Information Processing Systems, 36:51008-51025 2023.
15

---

Published as a conference paper at ICLR 2026
Christopher Williams, Fabian Falck, George Deligiannidis, Chris C Holmes, Arnad Doucet, and Saifuddin Syed. A unified framework for u-net design and analysis. Advances in Neural Infor- mation Processing Systems, 36:2775--777777722222 2023.
Weihao Xia, Raoul De Charette, Cengiz Oztireli, and Jing-Hao Xue. Dream: Visual decoding from reversing human visual system. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 8226-88 8035, 2024.
Xingqian Xu, Zhangyang Wang. Gong Zhang, Kai Wang, and Humphrey Shi. Versatile diffusion: Text, images and variations all in one diffusion model. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision, pp. 7754-7765, 2023.
Chengrun Yang. Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoe Le, Denny Zhou, and Xinyun Chen. Large language models optimizers. arXiv preprint arXiv:2309.033000 2023.
Ling Yang. Zhaochen Yu, Chenlin Meng, Minkai Xu, Stefano Ermon, and Bin Cui. Mastering text- to-image diffusion: Recaptioning, planning, and generating with multimodal 1lms. In Forty-first International Conference on Machine Learning. 2024.
Jacob Yeung, Andrew Luo, Gabriel Sarch, Margaret M Henderson, Deva Ramanan, and Michael Tarr. Reanimating images using neural representations of dynamic stimuli. In Proceedings of the Computer Vision and Pattern Recognition Conference, pp. 5331-5343, 2025.
Mert Yuksekgonul, Federico Bianchi, Pratyusha Kalluri, Dan Jurafsky, and James Zou. When and why vision-language models behave like bags-of-words, and what to do about it? arXiv preprint arXiv:2210.01936 2022.
Lvmin Zhang. Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. In Proceedings of the IEEF/CVF international conference on computer vision, pp. 3836-3847, 2023.
Richard Zhang, Phillip Isola, Alexei A Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE conference on computer vision and pallern recognition, pp. 586-595, 2018
Tianjun Zhang, Xuezhi Wang, Denny Zhou, Dale Schuurmans, and Joseph E Gonzalez. Tempera: Test-time prompting via reinforcement learning. arXiv preprint arXiv:2211.11890 2022.
Yuyao Zhang, Jinghao Li, and Yu-Wing Tai. Layercraft: Enhancing text-to-imnge generation with cot reasoning and layered object integration, 2025. UR//tttps.//axxi..orr//bbbb/
00010
Yongchao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. In The Eleventh Interna- tional Conference on Learning Representations, 2022.
Yixin Zhu, Yibiao Zhao, and Song Chun Zhu. Understanding tools: Task-oriented object modeling. learning and recognition. In Proceedings the IEEE Conference on Computer Vision and Pattern Recognition, pp. 1355-2864,, 2015.
Regine Zopf. Marina Butko, Alexandra Woolgar, Mark A Williams, and Anina N Rich. Repre- senting the location of manipulable objects shape-sclective occipitotemporal cortex: Beyond retinotopic reference frames? Cortex, 106:132-150, 2018.
16

---

Published as a conference paper at ICLR 2026
A A COMMON ERROR IN GENERATIVE MODELS
Original Image
Generated Image
Figure 3: A Common Error in Generative Models. While the original image shows "a gray tiger- striped cat," the model incorrectly generates "a grey tiger," illustrating semantic distortion.
In this section, we present a common failure, attribute binding, encountered in generative models (especially for the diffusion model), where generative models misattribute visual properties ob- jects. Figure 3 compares original (left) and generated (right) images. The original depicts a gray tiger-striped cat on wooden bench, while the generated version incorrectly shows gray tiger stead of a cat. This issue arises because diffusion models usually rely on text encoders such as CLIP, which are known lack the ability to capture complex linguistic structures (Yuksekgonul et al., 2022). Consequently, the diffusion process loses awareness of the bindings between objects and their attributes, leading to mismatched visual properties. This impairs fMRI-to-image reconstruc- tion. To address this, we introduce neurosciennce-issiree. object-centric generation approach that improves reconstruction quality.
B PRELIMINARY: DIFFUSION MODEL
Diffusion models are class of generative models that synthesize data by reversing gradual noising process. Given a data point Hy (e.g., an image), the forward process perturbs into Gaussian noise over T time steps. The model then learns the reverse process to reconstruct samples from noise. The forward process is Markov chain defined by:
q (H, H,-1) (H; V1 H, t= t=
where {B:)²= predefined variance schedule. The model is trained to predict the noise added to the input, using neural network eg, by minimizing:
C = EHo.e, [|le - c (H,,t))1²
Here, H, = Vo Ho + - with ã₂ = I1 - Bz), and e ~ I). To guide the gen-
eration process with external information C (e.g., prompt), the denoising network is extended as:
cØ (,,,t
Then, the training objective becomes:
Leond E ||e (H,,,, ]
17

---

Published as a conference paper at ICLR 2026
This formulation is widely used in text-to-image diffusion models, where C is the embedding of textual description obtained from a pre-trained text encoder (e.g., CLIP). In practice, the condition C incorporated into the U-Net via cross-attentin modules.
In our work, we adopt the latent diffusion framework (Rombach et al. 2022), where the diffu- sion process is applied in the latent space of pre-trained VAE, rather than directly in pixel space. Specifically, an input image Y E R#XW x3 is first encoded by a VAE encoder into a compact latent representation Z phxuxe.
Z Encoder(YY).
The diffusion process is applied on Z, where the perturbed latent representation Zr is obtained after steps. The reversed denoising steps then generate a denoised latent 2 over T steps. The final image is reconstructed by the denoised latent:
Y Decoder
This formulation greatly reduces computational cost while maintaining high-quality image genera- tion and is particularly well-suited for conditioning on high-level semantic representations such as text or fMRI-derived embeddings.
C DATASET
In this subsection, we provide information about the three pre-processed datasets used for the fMRI- to- to-image reconstruction task: NSD (Allen et al. 2022), BOLD5000 (Chang et al., (2019), and GOD
Horikawa & Kamitani 2017).
NSD (Allen et al. 2022): The Natural Scenes Dataset (NSD) is a large-scale public fMRI dataset capturing brain responses of human participants viewing naturalistic stimuli from COCO images (Lin et al.) 2014]. The dataset includes scans for 30-40 hours across 30-40 separate sessions. During each session, participants viewed 750 images for seconds each. Each image was presented three times across sessions, with most images unique to each subject, except for 1,000 shared images seen by all subjects. Following prior NSD recon- struction studies (Scotti et al. [2023] Takagi & Nishimoto (023), we adopt the standardized train/test split, where the shared images serve the test set. Consequently, the training set for each subject contains 8,859 image stimuli and 24,980 fMRI trials, while the test set includes 982 image stimuli and 2,770 fMRI trials.
BOLD5000 (Chang et al., 2019: The BOLD5000 dataset is a publicly available fMRI dataset capturing brain activity subjects viewed series of images. It contains 4,916 unique images, including 2,000 from the COCO dataset and 1,916 from ImageNet (Deng et al. 2009). Each image was presented as a visual stimulus in individual trials. Of these, 4,803 images were shown once, while 113 images were repeated three or four times across trials, resulting in a total of 5.254 stimulus trials. We follow the standardized train/test split used in prior BOLD5000 reconstruction studies (Chen et al.) 2023 Wang et al.) 2024c). Specifically, the training set includes trials with non-repeated image stimuli, comprising 4,803 samples, while the test set consolidate repeated image stimulus trials into 113 sam- ples.
GOD (Horikawa Kamitani 2017): The Generic Object Decoding (GOD) public dataset developed for IMRI based decoding. It aggregates fMRI data gathered through the presentation of images from 200 representative object categories, originating from Ima- geNet. We follow the standardized train/test set split employed in existing GOD image reconstruction studies (Sun et al. 2023) and get 1200 training samples and 50 test sam- ples. Generic Object Decoding (GOD) dataset is publicly available fMRI dataset de- signed for decoding object representations.
includes fMRI data collected during the presentation of images from 200 representative object categories sourced from ImageNet. Following the standardized train/test split used in prior GOD reconstruction studies (Sun et al. 2023), we use 1,200 training samples and 50 test samples.
18

---

Published as a conference paper at ICLR 2026
D BASELINES
In this section, we provide details about the baselines used in our experiments.
Takagi & Nishimoto (Takagi & Nishimoto 2023 : This baseline maps fMRI signals to the latent space of a pre-trained VAE within diffusion model using linear regression, en- abling image reconstruction. The method combines image latent representations with text embeddings extracted from CLIP text encoder, both mapped from fMRI signals in higher (ventral) visual cortex regions, to improve reconstruction quality. For fair comparison, adapt this approach to the Diffusion 2.1 pipeline by retraining the linear regression to map IMRI signals to both the VAE latent space and the textual conditioning used in Diffusion 2.1.
Mindvis (Chen et al. 2023): This baseline uses a self- ceff-sueeeeiee representation of fMRI data using masked modeling within a high-dimensional latent space an decoder framework. The learned representation is then projected into the conditioning space of LDM by fine-tuning the model. For fair comparison, we adapt the image generator of this approach to Diffusion 2.1 by fine-tuning it with the learned projection module, following the strategy outlined by the original authors.
Mindeye (Scotti et al. 2023 :: This model proposed two modules to map the fMRI signal to the CLIP image space. Specifically, the model first uses contrastive learning to align fMRI signals with image embeddings. Second, the paper trains a diffusion prior to reconstructing images from these embeddings via mapping brain activity into CLIP image space, enabling the generation of images that closely resemble the original stimuli. To adapt the method for fair comparison, we replace the Versatile Diffusion with Diffusion 2.1.
MindBridge (Wang et al. 2024b): This approach unifies heterogencous voxel dimensions using adaptive max pooling and employs a cyclic fMRI reconstruction mechanism to align neural responses across subjects in a common semantic space. To adapt the method for fair comparison, we replace the Versatile Diffusion with Diffusion 2.1.
NeuralDiffuser (Li et al. 2025): This paper proposes a visual feature-guider reconstruc- tion method that decodes multiple layers of CLIP's visual encoder from fMRI and uses these features as gradient-based cues during the early stages of reverse diffusion, enabling bottom-up detail refinement. It further introduces momentum alignment to reduce the dis- tribution shift between training and testing embeddings. For fair comparison, we use Dif- fusion 2.1 as the generative backbone.
Mindeye2 (Scotti et al., 2024): This method trains multiple MLPs to project fMRI signals from all subjects into shared representation space, followed by training a diffusion prior to map these representations into the CLIP image embedding space. The final image is then reconstructed using a pre-trained SDXL (Podell et al.) (023). To adapt this method to our setting, we replace the generative backbone with Diffusion 2.
E TRAINING DETAILS
In this section, we provide the training details of our model. Our model is implemented with Pytorch and trained on two NVIDIA-L40 GPUs with 48GB of memory. We use T5 as the language model to generate the object-level descriptions. For NSD data, we train the model for 80 epochs: 60 epochs for MLP training (EMLP) with learning rate It, 1 10 followed by 20 epochs of joint training, where we continue training the MLP and fine-tune the T5 model (Ers) using a learning rate lt, 5 1 For BOLD5000, we set Empi 50, Ir, 1e 5, Ers 5. and Ir, 10-8
For GOD, we set Empp = 40, Ir le - 5, ETs- = 5. and Ir2 5e 9 For image reconstruction at inference time, we set the blending ratio 3 = 0.5 and the denoising step as 40 for all the datasets. We use GPT 4o-mini to generate the object-centric descriptions with the prompt shown in Section For images that do not have a caption, we first use then use our prompt generate the object-centric description.
to generate a short caption and
19

---

Published as a conference paper at ICLR 2026
Table 7: Reconstruction performance across three latent spaces. The best result each column is highlighted in red. 1 indicates higher is better and | indicates lower is better.
PixCorr 1
SSIM
LPIPS L
CLIP
Inception V3 f
BOLD5000
Ours
0.231540.01
0,534 +0.02
0.6198
0.7720
0.6601 +0.07
Clip text
0.2000
1.4885+0.00
0.6520+0.04
0.7265+0.06
0.619940.00
LDM
0.1622+0.03
0.4300
0.7894
0.7025
0.5590 #0.05
GOD
Ours
0.2571 +0.01
0.5200
+0.02
0,6213 =0.01
0.8567
20.05
0,8428
10.06
Clip text
0.2200 +0.05
0.4682 =0.04
0.6827 =0.04
0.8120 #0.05
0.7602 #0.04
LDM
0.1900+0.02
0.399940.00
0.7100
0.7099
0.7484 #0.03
F CASE STUDY ON OBJECT-LEVEL DESCRIPTIONS FOR IMAGE RECONSTRUCTION
Generated Text
Visualiaation
Feed text generative model
Cars:
drive along The cors foll
Reconstructed
Latent representation
Image
belind the elephant's path.
Viewed Image
EMRI
Latent representation comcuteration within generative model
Encoder
Text preprucess
Elepkant:
Latenal representations
elephant walks down the road alone. The elephant faces from the cars.
Feed text generative madel
Vinualization
Figure 4: Reconstructed images using object-level descriptions.
In this section, we present a case study illustrating how object-level descriptions are used during the reconstruction process, as shown Figure 4 In the given example, the generated text is seg- mented based object instances: segment describes the cars ("The cars drive along the curved road...") and the other describes the elephant ("The elephant walks down the road alone...") Each object description is independently fed into diffusion-based generative model to produce object- centric latent representations. These representations are then spatially aligned and concatenated within the generative model to produce the final reconstructed image. This modular process helps mitigate common generation errors, such as attribute binding.
G RECONSTRUCTION PERFORMANCE ACROSS DIFFERENT LATENT SPACES
We report the ablation studies to justify our choice of text as the latent space on BOLD5000 and GOD. As shown in Table 7 aligning fMRI signals to the language model text space (our method) consistently outperforms alignment to the other two spaces across all metrics. This supports our core contribution: textual representations alone sufficient capture both high-level semantic and low-level visual information, and text space provides a more brain-aligned and effective intermediate representation for fMRI-to-image reconstruction.
20

---

Published as a conference paper at ICLR 2026
Table 8: Comparison of the number of objects in our framework. Results reported using PixCorr, SSIM, LPIPS, CLIP, and Inception V3 metrics. The best result in each column is highlighted in red. indicates higher is better and indicates lower better.
P
SSIM f
LPIPS 4
CLIP 1
Inception V3 f
Ours (Two Objs)
0.3404 #0.05
0.464 #0.02
0.5943 #0.02
0.9467
0.9516 #0.03
One Obj
0.3355 10.05
0,4532
10.04
0.6014 10.04
0.9344
10.02
0.9342 10.01
Four Objs
0.3202 #0.03
0.4469 +0.01
0.6284 =0.02
0,9400 +0.05
0.9322 #0.05
H ANALYSIS ON THE NUMBER OF OBJECTS IN OUR FRAMEWORK
In our framework, we fix the number of objects image to two and assign separate MLP to each. We learn to assign each object location label from predefined set of spatial positions (e.g., left/right or top/bottom). This fixed assignment of each object to a dedicated MIP, along with the predefined spatial labeling scheme, is used consistently during both training and inference. At inference time, each MLP independently encodes fMRI signals for object, and the language model generates a structured description for each. These descriptions are then passed to the object- centric diffusion model, which generates object images independently and places them into their corresponding spatial positions to form the final image.
We conduct experiment determine the optimal number of objects (MLPs) in framework, and the results on the NSD dataset are reported in Table The results show that setting the number of objects per image m = and m = in Figures those containing four objects), setting m encourages the VLM to describe the first object group of similar instances, rather than generating isolated descriptions for each object. As shown Figure 5 the individual players are grouped and described as "three players." " In contrast, setting m 4 prompts the VLM to generate object-level descriptions (e.g., generating one description per player, as shown Figure leading the generative model to produce distinct latent representations for each object. These representations are then concatenated by spatial location and passed to the pre-trained diffusion model. However, when too many object-level features are introduced, diffusion model tends to omit objects during the denoising process Liu et al. (2024); Wang et al. (2023), resulting in incomplete reconstructions- for example, with one of three players missing in the final output (Figure
yields the best performance. We further present
studies for setting respectively. We observe that for more complex images (i.e.,
Viewed Image
Generated Text
The Frisbee:
The frisbee
Feed The
Frisbee' text to generative model
floating above the players bright
green outdoor
IMRI
Encoder & LM Text preprocess
Image representation concatenation within generative model
Three players: The middle player is jumping up with arms stretched toward the frisbee. The other players are trying to catch the frisbee
Feed Three players' text to generative model
Figure 5: Reconstructed images using two objects.
21

---

Published as a conference paper at ICLR 2026
Generated Text
Viewed Image
Feed the frisbee' generative model
The Frisbee:
The frisbee is flying above the players
One Player: The player
Feed One player' text to generative model
FMRI
standing white clothes bright green outdoor.
Encoder & LM Text preproeess
Image representation concatenation
Middle player: The Middle Player stands directly under the frisbee. blue shirt and black athletic shorts
Feed Middle player's 's text to generative model
Feed Right player's 's text to generative model
Right player: The left player runs toward the frisbee
Figure 6: Reconstructed images using four objects.
I ATTRIBUTE/RELATIIONNS OPTIMIZATION DETAILS
The detailed algorithm used solve the prompt optimization problem in Equation (3) is provided in Algorithm In our experiments, we adopt the scoring function S(Y,, Y2) = (PIPS(Y,, Y2)- The CKA threshold is initialized the minimum CKA score among the initial candidates. We set & = 0.5, k1 = 8, K2 2 and search for T = 40 rounds. We randomly sampled 667 images from the training set of NSD (Allen al. 2022) for prompt optimization. While this subset used for efficiency, our method is applicable to the full training set and generalizes to other settings. We use GPT-4o minn as the VLM and the LLM that generates new keywords. We use CLIP-text (Radford et al. (021) LMENG, and Stable Diffusion 2.1 (Pernias et al. 2023 Rombach et al. 2022 as the diffusion model.
Algorithm e-Greedy Prompt Optimization
Input: Training set A yyrain Initial keyword set A: Search rounds T; Parameters E, k l2 Initialize: Threshold 3 <- minae CKA (X, K
for
Filter: A < {a A CKA (X, B} Sort: Rank a e in descending order by ΣN SY, Dif((VLM(Y;; P(a)))
if random () < ı then
Sample: < RandomSample(A4 k.) % Randomly sample k keywords from A
else
Select: +- Top(AAAi)
Select top-ki keywords from A
end if
Generate: Use LLM to synthesize K2 new keywords Anow based on S
Update: A < A Anew
end for
Output: arg m EN SY,, DII(VLM(YY
The search is initialized with six widely-used keywords describing object attributes and relation-
ships: Semantic Relationship (Johnson et al.] 2015). Positional Relationship (Lu et al. 2016
Haldekar et al.] 2017), Functional Attributes (Zhu et al. 2015), Action Attributes (Lu et al. 2016), Visual Attributes (Farhadi et al., 2009), and Part- Whole Relationship (Lu et al. (016)). For each type, GPT-40 generate four synonymous keywords, resulting in an initial pool of 24
candidate keywords. Figure Ingureereppnr reports the LPIPS scores of all initial and subsequently discovered keywords.
22

---

Published as a conference paper at ICLR 2026
J KEYWORD SEARCH WITH DIFFERENT INITIAL SETS
To further demonstrate the effectiveness of our atrribute/relaioonhipp search module, we conduct an additional experiment where we excluded spatial-related keywords from the initial keyword set. We then evaluated whether our search procedure could still find spatially relevant terms. As shown in T the module successfully identifies spatial keywords under this constraint, demonstrating robustness and effectiveness of our search process.
Table 9: Top-5 discovered keywords across search rounds.
Rank
Round 0
Round 10
Round 20
Round 30
Textural Attribute
Spatial Relationship
Spatial Arrangement
Spatial Arrangement
#2
Conceptual Linkage
Textural Attribute
Proximity Dynamics
Proximity Relation
#3
Action Interaction
Visual Harmony
Spatial Relationship
Proximity Dynamies
#4
Chromatic Feature
Emotional Resonance
Color Arrangement
Spatial Relationship
Autribute Description
Conceptual Linkage
Hue Contrast
Color Arrangement
K PROMPTS USED BY PRISM
Prompt in generating structured descriptions
Given the image and caption, first describe the background color style of the image with 3-5 words. Second, detect the TWO most important objects in the image. Then, describe each of the objects and their relationship using: [keyword] with TWO sentences. For each sentence, use 5- 10 words and as easy as possible.
Then, detect the absolute position of the two objects the image, and select from [right, left, top, bottom). "left" and "right" should appear together for horizontal objects, and "top" and "bottom" should appear together for vertical objects. DO NOT mix.
Example:
### Background color style: Grayscale urban.
### The Man [left]
1. The man is standing near the sidewalk edge. The Man is close to the building wall.
### The Suitcase [right]
The suitcase is beside the man's foot. The Suitcase is placed on the street's curved edge.
Now, given the image uploaded and the caption "(caption)", detect the two most important objects with absolute position, describe them using (keyword) with EXACTLY the example format:
Prompt in attribuute/relationsiip optimization
System prompt: You are a helpful brainstormer. Given a list of keywords, generate (gen_nuum) related or similar keywords. Respond with a comma-squae conma-scparated list of keywords.
User prompt: keyword: (keywords]
23

---

Published as a conference paper at ICLR 2026
Spatial Layout
Spatial Pattems
Spatial Organization tion
Relative Position
Spatial attorshipss
Structud
Positional Context
Spatial Distribution
Spatial Reiationship
Spatial Arraegement
Spetial Configuration
Spatial Interaction
Spatill Connectvity
Spanil Hicrarchy
Spatial Analysi
Positional Reia lion
Spatial Dynamics
Festure Rela tion
Geometric Relationships
Structural Rolationship
Structural Hierarchy
Hlerarchical Structure
WholePart Reiat Relationship
Location Rela tion
Relational Dynamics es
Enclesure Roiation
ipscrritiveeee Attribute
inclusion Dependency
Interaction Context
Relational
PartWhole Relationship
Containment Strategy
AttributeReeationsst Keywords
Action Connection tion
Structural Framework
Boundary Definition
Conta containment Structure
Whote Rotation
PartiWholc Dynamics cs
Conceptual Cornection tion
Contexitual Reiationship
Feature
Action tion
Semantic Associa tion
Visual Attribute
Contextual Dynamics
Contextual Associatio tion
Agent Interaction
Functional Dependency
Behaviaral Dependency
Cognitive Mapping
Feature Connectio tion
Affordance Connectio tion
Visual ture
Semantic tion
Visual Represanta
Contextual Affordance
interaction Dynamics
Visual Trait
Functional Interaction
Affordance Rela tion
Semantic Connection
Meaning Link
Dynamic Intoraction
Visual Context
Object Engagement
Functional abonship
Qualitative
Usage Relation
Structural Dynamies
Function Connection tion
Componentinlegre tion
Geometric Link
Usage Context
Contextual Connection
Cognitive Connection
Affordance Intoraction
Behavioral Dependence e
interactive Dynamics es
Objest Interaction
Impact-
Containment System
Property Link
Object Affordance
Behavioral Influence
Purpese Relation
Affordance Rejationhipp
Conceptual Link
Semantic Link
Functional Affordance
Initial Words
Functronal Design
Newly Generated Words
User Experience
0.3400
0.3425
0,3450
0,3475
0,3500
0.3525
0.3550
0.3575
0.3600
Scores (1-LPIPS)
Figure 7: Scores of all keywords during the prompt search. Among the top- 10 keywords, the most frequent keyword is 'spatial'.
24

---

Published as a conference paper at ICLR 2026
L THE USE OF LARGE LANGUAGE MODELS (LLMS)
We declare that Large Language Models (LLMs) were confined to peripheral tasks and had no in- fluence on the methodology, results interpretation, theoretical insights of this work. Specifically, they were used for (i) generating training datasets required for our experiments and (ii) grammar correction and minor word-level refinements. All language edits were carefully reviewed by the au- thors to ensure that no hallucinations were introduced and that the text faithfully reflects the original intent. The technical development, experimental design, analysis, and conclusions presented here are entirely the work of the authors.
25


Reviews:
[{"review_id": "rnDdJyXGjE", "reviewer": "Reviewer_M3K1", "summary": "The reconstruction of visual stimuli (images) from functional Magnetic Resonance Imaging (fMRI) signals involves a two-stage pipeline: transforming fMRI signals into a latent space and then using a pre-trained generative model to create the final image. The quality of the final reconstruction is largely determined by the alignment between the latent space and the structure of the neural activity, as well as the generative model's efficiency in producing images from that space. The proposed work demonstrates that accurate visual stimuli reconstruction can be achieved using the text space of a Large Language Model (LLM) instead of image-based latent representations. Furthermore, the reconstruction quality is further improved by object-centric diffusion that generates images by composing individual objects and identifying object attributes and relationships within the neural activity captured by the fMRI signals. Experiments on real-world datasets demonstrated that the proposed PRISM framework outperformed existing methods in CLIP metric by (in absolute terms) by 0-4% and in Inception V3 metric by 1-3%.", "questions": "Please refer to the weaknesses.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper makes an interesting observation that fMRI signals exhibit greater similarity to the text space of a language model than to either a vision-based space or a joint text-image space.\n\n2. The pipeline for generating images that captures object relationships in the text space is intuitive.\n\n3. The paper performs better than existing methods on many datasets, thus showing the importance of utilizing a structured text space as the an intermediate bridge between fMRI signals and image reconstruction.", "weaknesses": "1. The experimental results need to be clarified. How is the result for MindEye2+SDXL achieved? The original paper itself shows worse performance. The details regarding which NSD subject is used need to be provided.\n\n2. Some other experimental comparisons need to be made:\n[1] Wang, Shizun, et al. “Mindbridge: A cross-subject brain decoding framework.” Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n[2] Gong, Zixuan, et al. “Mindtuner: Cross-subject visual decoding with visual fingerprint and semantic correction.” Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 39. No. 13. 2025.\n\n3. Results on other text spaces such as RoBERTa should be provided.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "7t0CngeyRi", "reviewer": "Reviewer_FGs4", "summary": "This paper introduces PRISM, a new framework for reconstructing visual images from fMRI signals. The authors find that fMRI data aligns more closely with structured text representations than with vision-only or joint vision-language spaces. The proposed method first decodes fMRI signals into an object-centric text description, which details objects, their attributes, and their relationships. This structured text then guides a specialized generative model to create the image, a process designed to improve the compositional accuracy of the reconstruction. The paper reports that this approach outperforms existing methods on three standard fMRI datasets.", "questions": "- The paper demonstrates that fMRI signals align better with the T5 language model's text space than with the tested vision or joint-latent spaces. How dependent is this key finding on the specific choice of models? For instance, would this alignment hold if compared against different, perhaps more recent or semantically robust, vision-only representations?\n\n- The entire training process for the fMRI-to-text encoder relies on structured descriptions generated by a pre-trained Vision-Language Model (VLM). How do errors, omissions, or inherent biases in this VLM's descriptions (which serve as the \"ground truth\" text) propag\n\n- The object-centric diffusion module is designed to improve compositional accuracy. How does this method scale when reconstructing highly complex scenes containing many objects, significant overlap, or abstract relationships that are inherently difficult to capture in the structured text format?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The strengths include proposing a novel approach that challenges the assumption of using vision-based representations, instead providing evidence for the effectiveness of a structured text space for fMRI decoding. The method introduces specific components, such as an object-centric generative module and an attribute-relationship search, to directly address the known problem of compositional accuracy in image reconstruction. The paper's claims are supported by comprehensive quantitative evaluations across three different public datasets, showing improved performance over existing baseline methods, and are further validated by targeted ablation studies.", "weaknesses": "- The paper's central claim that fMRI signals align better with \"text space\" than \"vision space\"  rests on a specific comparison. The \"text space\" is generated using embeddings from image captions, which are already highly semantic, human-generated descriptions of the images. This is contrasted with the latent spaces of vision models (e.g., LDM, ResNet50). Consequently, the finding may be more precisely interpreted as fMRI signals aligning well with semantic, object-centric descriptions of images, rather than with language or text in a more general sense. \n\n- The ablation study in Table 5 evaluates the impact of the \"Attribute-Relationship Search\" module. While the full PRISM model outperforms the \"w/o AttOpt.+Bst\" variant (which uses the best-performing initial keyword) , the performance difference between these two is smaller than the difference between the \"best initial\" and \"worst initial\" keywords. This suggests that while the search module adds value, a significant portion of the performance might be achievable by simply selecting a good, spatially-focused keyword from the initial set.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "nZJCgC7qfO", "reviewer": "Reviewer_jrM3", "summary": "I have finished reading the entire manuscript. This paper focuses on the task of reconstructing the visual images subjects once viewed from fMRI data. Unlike previous research, this paper did not choose the widely selected T2I models for reconstruction, such as SDXL, but instead opted for a class of object-centric T2I models. The authors believe that using object-centric generative models can better reconstruct information about the objects, their positions, and attributes within the visual scene.\n\nAt the methodological level, the authors use a pre-trained VLM to generate object-level structured descriptions for the images. To improve the accuracy of these structured descriptions, the authors designed a prompt optimization method. Subsequently, the authors employ a prompt tuning approach, using fMRI as input to an LLM to predict these structured descriptive languages. Finally, the predicted structured descriptions and the pre-trained object-centric generative model are utilized to reconstruct the seen image.\n\nThe paper's evaluation primarily utilized the NSD dataset, focusing on the accuracy of the synthesized images, and simultaneously included an ablation study of the method.\n\nIn my opinion, the innovation and contribution of this paper lie in the introduction of the object-centric generative model, which could enable more accurate reconstruction.", "questions": "+ The baseline methods, such as MindEye and MindEye2, used more reconstruction evaluation metrics, including Alex(2), Alex(5), Eff, and SwAV. Why did the authors omit these evaluation metrics?\n\n+ Based on the results in Table 6, this keyword optimization strategy does not seem to bring about a substantial improvement in the keywords. I believe that further evaluation and explanation are needed here.\n\n+ In the experiment, was the number of object-level structured descriptions, $m$, set to 2? This is a guess I made based on the prompt design on page 21. If so, I am curious whether extracting more objects would help with the reconstruction.\n\n+ According to Formula 4, is the same fMRI input repeated $m$ times to generate structured descriptions for a single image? What is the significance of this design? Alternatively, why is a dedicated MLP set up for each structured description?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+ The use of an object-centric generative model for more precise fMRI-to-Image reconstruction is novel.\n\n+ Evaluating the details of the reconstruction using a task similar to VQA (Visual Question Answering) is promising.", "weaknesses": "+ Although the authors demonstrate through experiments that aligning fMRI to the text representation space can yield better results, this approach is still unusual or requires further explanation. Based on the setup of the NSD dataset, the fMRI signals used by the authors originate from the nsdgeneral ROI, which is almost entirely the visual cortex of the brain. Therefore, this part of the fMRI data should logically contain only visual information, making the practice of embedding it into a pure text space seem to require further justification.\n\n+ The authors should introduce some more recent baselines for comparison. MindEye2 was published at ICML 2024, and more methods have emerged over the past year or so; the authors should discuss and compare against them.\n\n+ I believe that more case studies should be presented in the experimental section, including the object-level structured descriptions and their corresponding reconstructed images. I think this content would help to better understand the performance and effectiveness of the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GCkUxozmcK", "reviewer": "Reviewer_TY5P", "summary": "This paper challenges the prevailing assumption in fMRI-to-image reconstruction that the optimal latent space must match the visual modality of the stimulus. Through a systematic comparison, the authors present a key finding: the text embedding space of a language model shows stronger alignment with fMRI signals than vision or joint vision-language spaces. Building on this insight, the paper proposes PRISM, a novel framework that uses a structured text space as the intermediate representation.", "questions": "1. The authors should clearly state in the main text what the Encoder architecture is and which specific Large Language Model (LLM) is used for language decoding.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. The paper's central discovery—that fMRI signals align better with a text-based latent space than a vision-based one—is a significant conceptual breakthrough. This challenges a fundamental assumption in the field and provides a new and compelling direction for bridging neuroscience and generative AI.\n\n2. The authors conduct extensive experiments on three distinct fMRI datasets (NSD, BOLD5000, GOD) and show that PRISM consistently outperforms multiple state-of-the-art methods across a wide range of metrics. The inclusion of an image-based QA task further strengthens the claim that the reconstructions are not only visually faithful but also semantically meaningful.", "weaknesses": "1. The framework's core design assumes a fixed number of objects (m=2) per image. This is a significant constraint, as real-world scenes can contain one dominant object, or many. The neuroscientific justification based on cognitive load is an interesting hypothesis but also a simplification. This design choice may force the model to hallucinate a second object when only one is present or fail to capture the richness of a more complex scene, limiting its practical applicability and generalizability.\n\n2. The attribute-relationship search module is complex, involving an ɛ-greedy search guided by an LLM, constrained by CKA, and optimized for LPIPS. A critical baseline is missing: comparing PRISM's performance against a simpler approach where the structured descriptions are generated by a state-of-the-art vision-language model (e.g., GPT-4V) using a single, fixed, hand-crafted prompt. This would help isolate the contribution of the elaborate search mechanism itself.\n\n3. While the paper provides an ablation study in Table 8 showing m=2 is optimal, the analysis could be more thorough. For instance, does the optimal number of objects vary with image complexity or content? Providing examples where the m=4 model fails (e.g., by hallucinating objects) would make the argument for m=2 more concrete and visually intuitive.\n\n4. The case study in Table 6 demonstrates a convergence towards \"spatial\" keywords, which is an interesting result. However, the paper could provide a deeper neuroscientific interpretation of this finding. Why are spatial relationships more \"brain-aligned\" for reconstruction than, for example, functional or descriptive attributes? A more detailed discussion connecting these results to known properties of the visual cortex would enhance the paper's interdisciplinary value.\n\n5. The illustration in Figure 1 is not precise enough, making it difficult to intuitively grasp the authors' intended meaning by simply viewing the diagram. For instance, in the \"Attribute/Relationship Search\" section, the authors use only a single looping arrow to denote the complex iterative labeling process. Additionally, in the alignment between the \"Predicted Description\" and \"Structured Description,\" the sentence descriptions in both the left and right parts appear to be the same, which is confusing.\n\n6. While the paper employs extensive engineering optimizations to accurately decode objects and their relationships, it lacks sufficient neuroscientific interpretability. For instance, it does not provide a compelling explanation for its core finding: why brain signals map more strongly to language models than to vision models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Oral)