Towards Interpretable Visual Decoding with Attention to Brain Representations
Authors: Pinyuan Feng
Hossein Adeli
Wenxuan Guo
Fan L. Cheng
Ethan Hwang
Nikolaus Kriegeskorte
Venue: ICLR 2026 Poster (2026)
Keywords: Brain Decoding
Visual Reconstruction
Interpretability
fMRI
Latent Diffusion Model
Primary area: applications to neuroscience & cognitive science
TL;DR: We introduce NeuroAdapter, an end-to-end fMRI-to-image decoding method that directly conditions diffusion models on fMRI signals, achieving competitive reconstructions while enabling interpretable analysis of how brain regions guide generation.

Abstract: Recent work has demonstrated that complex visual stimuli can be decoded from human brain activity using deep generative models, offering new ways to probe how the brain represents real-world scenes. However, many existing approaches first map brain signals into intermediate image or text feature spaces before guiding the generative process, which obscures the contributions of different brain areas to the final reconstruction output. In this work, we propose NeuroAdapter, a visual decoding framework that directly conditions a latent diffusion model on brain representations, bypassing the need for intermediate feature spaces. Our method demonstrates competitive visual reconstruction quality on public fMRI datasets compared to prior work, while providing greater transparency into how brain signals drive visual reconstruction. To this end, we introduce an Image–Brain BI-directional interpretability framework (IBBI) that analyzes cross-attention patterns across diffusion denoising steps to reveal how different cortical areas influence the unfolding generative trajectory. Our work highlights the potential of end-to-end brain-to-image reconstruction and establishes a path for interpretable neural decoding.

Published as a conference paper at ICLR 2026
TOWARDS INTERPRETABLE VISUAL DECODING WITH ATTENTION TO BRAIN REPRESENTATIONS
Pinyuan Feng
Hossein Adeli'
Wenxuan Guo!
Fan Cheng!
Ethan Hwang!
Nikolaus Kriegeskorte*
Tuuckerann Mind Brain Behavior Institute, Columbia University, USA
Project Page
ABSTRACT
Recent work has demonstrated that complex visual stimuli be decoded from human brain activity using deep generative models, offering new ways to probe how the brain represents real-world scenes. However, many existing approaches first map brain signals into intermediate image or text feature spaces before guid- ing the generative process, which obscures the contributions of different brain areas to the final reconstruction output. In this work, we propose NewroAdapter, a visual decoding framework that directly conditions latent diffusion model on brain representations, bypassing the need for intermediate feature spaces. Our method demonstrates competitive visual reconstruction quality on public fMRI datasets compared to prior work, while providing greater transparency into how brain signals drive visual reconstruction. To this end, we introduce an Im- age Brain BI-directional interpretability framework (IBBI) that analyzes cross- attention patterns across diffusion denoising steps to reveal how different corti- cal areas influence the unfolding generative trajectory. Our work highlights the potential of end-to-end brain-to-image image reconstruction and establishes path for interpretable neural decoding.
INTRODUCTION
Prior Work
Our Work
Stage 1
End-to-End Decoding
Stimuli
Stimuli
Image
Decoded Stimuli
Stage
Interpretability Probing
Founi--onnatioon Model
Aligned Fenture Space
Decoded Stimuli
Image
Generator
Brain Surface Map
Weights
ROLAtta Map
Overview. Left: Typical two-stage decoding pipelines first map brain activity interme- diate feature spaces (e.g., CLIP/DINO) and then use those embeddings guide a generative model. Our end-to-end approach conditions latent diffusion model directly on brain activity, en- abling interpretations of the generative dynamics in both image and brain spaces.
Figure 1:
Right:
Understanding how the human brain represents the visual world remains a central challenge neu- roscience. Neural decoding approaches help address this challenge by inferring the content of the
"pf2477@ cu24mi.ccoum

---

Published as a conference paper at ICLR 2026
representation in different brain areas across the whole brain in response to complex stim- uli. In recent years, decoding models have achieved remarkable success across different perceptual modalities and intended movements, with many pipelines incorporating deep generative models. These works have pushed the NeuroAl frontier of reconstructing content or decoding "thoughts" from brain activity, bringing the prospect of "mind reading" closer to reality.
Current approaches to reconstructing visual stimuli from brain activity (Lin et al. 2022 Cheng
al. 2023; Takagi & Nishimoto, 2023; Ozcelik & VanRullen 2023] Scotti et al.) 2023 Li et al.
2025 Ferrante et al. 2025) typically implement a two-stage pipeline (Fig. | left): (i) brain activity
first mapped to intermediate image- or text- embeddings derived from large foundation models
(e.g. CLIP (Radford et al. 2021) and DINO (Caron et al., 2021 Oquab et al. 2023 Siméoni et al.
2025 ); (ii) these intermediate representations are then used to condition a visual generative model for stimulus reconstruction. Mapping brain data into an intermediate representation space leverages rich priors embedding spaces to improve reconstruction quality and has proved highly effective for reconstruction. However, the use of this intermediate representation can introduce an information bottleneck (Mayo et al. 2024; Shirakawa et al. 2025). with successful reconstruction of perceived stimuli depending on the alignment between neural representations and the embedding space. This intermediate step can also mask the effect of different brain areas on the final reconstruction, limiting the interpretability of the approach. In this work, we explore alternative approach (Fig. [ right) to two-stage decoding pipelines: conditioning latent diffusion models directly on the brain activity.
Contributions of our paper. Our contributions are as follows: (1) we propose NeuroAdapter, an end-to-end framework that learns parcel-wise embeddings from fMRI data and integrates them into latent diffusion models through cross-attention; (2) we show that our approach achieves competitive performance on public fMRI datasets, demonstrating that high-quality visual reconstructions can be obtained without reliance on external embedding spaces; and (3) we provide a bi-directional inter- pretability framework, namely IBBI, which leverages cross- coss-attention dynamics across diffusion steps to reveal both the relative contribution of brain parcels and their spatial influence in the reconstructed images, offering new insights into the generative process from a neuroscientific perspective.
2 RELATED WORK
Brain Decoding with Deep Generative Models. Early pioneering work demonstrated that fMRI signals could be decoded into continuous visual experiences by treating reconstruction as a stimulus identification task. For example, Nishimoto et al. (2011) used motion-energy encoding model and Bayesian inference to retrieve viewed movie clips from a large library of candidates. With the rise of deep generative modeling, decoding has progressed from classification to photorealistic reconstructions that leverage powerful image priors. Early GAN- pipelines established the feasibility of mapping brain signals into deep feature spaces and synthesizing images (Seeliger et al., 2018 Shen et al.) 201 9019a| b. Cheng et al. 2023 Gu et al. 2024). Latent diffusion has since become the dominant image prior, with several methods steering Stable Diffusson via IMRI-predicted image/text
latents (Lin et al. 2022; Chen et al. 2023 Ozcelik & VanRullen 2023 Scotti et al. 2023
Takagi &
Nishimoto 2023 Zeng et al. 2024 Wang et al. 2024b).
Recent studies have experimented with different conditioning inputs, training regimes, cross- subject alignment strategies (Xia et al. 2024] Han et al.) 2024, Huo et al. 2024 Li et al. 2025 FFerrate Ferrante al. (2024) has shown that aligning NSD subjects' fMRI into a shared functional space enables cross-subject reconstruction. Despite this progress, most pipelines still route brain activity through intermediate vision or vision-language feature bottlenecks to guide generations. The latest streamlined approach, Dynadiff (Careil et al.)
Wang et al., 2024a| Gong 2025). In particular,
2025 moved towards single-stage solution by using LoRA finetuning (Hu al. (002) for dy- namic visual decoding from time-resolved fMRI signals. In contrast, our proposed NeuroAdapter conditions the latent diffusion model directly on brain representations via cross-attention enabling more transparent and anatomically grounded interface between fMRI signals and the generative
model.
Interpretable Visual Decoding. A central goal of visual neuroscience is to understand both the functional selectivity of brain areas (what information they encode) and the representational formar of that information. Encoding approaches advance the first goal by learning brain encoder that
2

---

Published as a conference paper at ICLR 2026
IMRI Collection
Parcellation
Parcel-wise Linear Mapping
Left
Mapping Module
Right
Stimuli Subject
[MRI Duta
(d) Brain Decoding with Stable Diffusion
Cross Attention Module
Moo o0l
IMRI Token Dropout
Trainable Modules
Decoded
Latent 70
Repeat for L-1 times
7--1 Denoising U-Net
Z1
Stimuli X" Decoder
Frozen Modules
Reversed Diffusion Process
Figure 2: NeuroAdapter training pipeline. (a) fMRI data collection paradigm, (b) cortical parcel-
lation, (c) parcel-wise linear mapping from vertices to brain representation tokens, and (d) condi- tioning a latent diffusion model on these tokens for reconstruction.
maps images neural activity, and then using this encoder to (i) optimize stimuli that maximally drive a given cortical region (Luo et al. 2023a) or (ii) generate natural-language descriptions of voxel-level selectivity (Luo al.]| (023b)) Complementarily. transformer-based brain encoders pro- vide an interpretable architecture whose attention maps explicitly route visual features into distinct brain areas (Adeli et al. 2025), offering mechanistic insight into functional organization (Hwang it al. 2025). In contrast, decoding approaches target the second goal by testing what can be read out from neural activity and how reconstructions depend on specific regions, thereby probing the format and distribution of visual information. Studies that train and test decoders on subsets of visual areas have revealed how information distributed across the visual hierarchy (Shen et al.)
2019a by Horikawa & Kamitani 2022 Cheng et al. 2023 Ozcelik & VanRullen] 2023). Paral- lel developments in language neuroscience introduce interpretable embeddings and causal testing frameworks to link representational dimensions to brain activity (Tang et al. 2023a Benara et al.
2024 Antonello et al. 2024).
A key scientific motivation for decoding, alongside encoding analyses, is that they address comple- mentary questions. Encoding models characterize how external stimuli are transformed into neural responses. Decoding instead asks what aspects of visual mental content can be reliably read out from measured neural activity, which is particularly important in settings where the relevant sub- jective percept is only partially constrained e.g., visual illusion (Cheng et al., 2023), mental imagery (Kneeland et al. (202)), dreams (Horikawal al., (2013), and other forms of subjective perception. To leverage decoding for scientific insight, it is essential to understand how a decoding model uses brain signals to guide image generation. Existing analyses of latent diffusion models examine when (diffusion time step) and where (model layer) low- and high-level features emerge in the model (Takagi & Nishimoto 2023), but typically lack dynamic view of which parts of the generated image are modulated by brain-derived infor- mation. Our work addresses this gap by using cross-attention to provide explicit, temporal maps linking brain signals to image regions throughout the denoising process.
cannot be fully specified by external stimulus,
3 METHODS: MODEL TRAINING AND EVALUATION
Our brain decoding model, NeuroAdapter, as shown in Fig. is built the IP-Adapter framework Ye et al. 2023). We conditioned a pre-trained Stable Diffusion model (SD: Rombach et al. (2022) ) on deriiee features via cross-attention mechanism reconstruct perceived visual stimuli. In this section, we explain the details of our method with the Natural Scene Dataset (NSD; Allen et al. 2022 but similar method applies to the other datasets as well.
https://hugging co/ stable e- fuss on -fussioonvll-//taallee.. ffusionnvl-11
3

---

Published as a conference paper at ICLR 2026
3.1 NEURAL DATA PROCESSING AND PARCELLATION
We trained our model using the basece-based fMRI data in fsaverage space. We first averaged the vertex responses across image repetitions to obtain a single response pattern per image. To transform the high-dimensional fMRI data into structured inputs for conditioning the diffusion model, we applied the Schaefer parcellation ((Schaefer et al., (017);; see Appendix (C). This clusters cortical vertices into 500 parcels per hemisphere and has been shown to be an effective practice for brain tokenization (Bosch et al., 2025).
To improve robustness of the model by restricting inputs high-quality regions, computed vertex- wise Signal-to-Noise Ratio (SNR) and selected top & parcels per hemisphere with the highest average SNR, yielding total of p 2k parcels fMRI conditioning inputs the model. In the following sections, we report results of our model trained on 200 brain parcels, and present an ablation study on how varying p influences decoding performance in the Appendix
3.2 PARCEL-WISE LINEAR MAPPING
Since the number of vertices varies across parcels, we padded each parcel's vertex response vector match the largest vertex count across parcels Umaz. This yields processed neural data DIMRT Raxpxtmn where is the batch size of stimulus images. Then, each parcel was assigned a unique projection matrix w E yun xf, transforming padded vertex response into fMRI token embeddings E E R=xxxx where f is the hidden dimension of fMRI token embeddings. In the main text, we set f 768 during model training, and results from an ablation study with different values of f' is provided in the Appendix H Additionally, we conducted another ablation study (Appendix []] to demonstrate that mapping fMRI data into the parcel- wiiee token space to condition the SD generation effective for visual reconstruction.
3.3 LATENT DIFFUSION PROCESS WITH BRAIN CONDITIONING
We replaced the cross-attention layer of the U-Net (Ronneberger et al. 2015) in SD with an IP- adapter-style cross-attention module YYe et al.] 2025), enabling the model to attend to the fMRI token embeddings. To ensure that embeddings were the only conditioning input, the text encoder in SD received an empty input during both training and inference. During training, only the parcel-wise linear mapper and the new cross-attention modules were updated, with the rest of the parameters kept frozen.
fMRI Token Dropout. We applied stochastic token dropout strategy during training to the fMRT token embeddings E to ensure robustness of visual decoding. We randomly dropped out parcel-wise token vectors for each training sample. A dropout probability r U(00 1) was drawn, and each [MRI token vector was independently retained with probability 1-r. This resulted in binary mask M E {0, 1]xxpx1, which was applied parcel-wiss to the fMRI token embeddings E' = O M. We found this regularization to be crucial for strong decoding performance, supported by the ablation results in Appendix G
Min-SNR Loss Weighting, To stabilize training and improve sample quality, we adopted the min- SNR weighting strategy (Hang al. (023) recently introduced in diffusion models. This approach doin-weights the contribution of easy high-SNR steps, where reconstructions are clean, while pre- serving the weight of noisy low-SNR steps, yielding a more balanced training signal across the diffusion process (please view Appendix M for details).
3.4 DECODED IMAGE SELECTION WITH BRAIN ENCODING MODEL.
Inspired by Kneeland et al. (2023), we used a whole-brain encoder (Adeli et al. 2023 2025 H Hwang et al. 2025 trained on the same fMRI-image training dataset to identify the best decoded stimuli during evaluation. As shown in Fig. 3 (a). for each fMRI sample in the test set, the decoder generated set of candidate images X ' X with different random seeds. The brain encoder predicted vertex- fMRI activity By ... B', for the candidate images, which was correlated with the ground-truth fMRI measurements. The candidate image with the highest Pearson correlation was
4

---

Published as a conference paper at ICLR 2026
Brain Encoder-based Image Selection
Brain Encoder Performance
X'o
Decoded Stimuli
Ground Truth
X'1
0.8
Peurson Carr
Ranking
Pearson Correlation
X'n
B'a
0.4
0.61
0=
0.55
0.46
0.59
0.49
S1
S2
SS
S7
Stimulus
Predictions
Subjects
Figure 3: Brain Encoder. (a) Brain encoder-based image selection using Pearson correlations between predicted and measured fMRI responses for NSD test example. (b) Red: correlation between the predicted brain activity from the decoded images and the measured brain activity. Blue: correlation between the predicted activity for the stimulus in testing set and the corresponding fMRI
response.
selected as the final decoded image for further evaluation. An ablation study assessing the impact of the brain encoder to decoding performance is reported in Appendix K
4 METHODS: IBBI FRAMEWORK FOR INTERPRETABILITY
Beyond decoding performance, we also investigated the interpretability of the generative process in our model. During inference, the SD model reconstructs images by progressively denoising latent representation over multiple steps, starting from pure Gaussian noise and gradually refin- ing toward a clean image. each denoising step t. the U-Net backbone applies a sequence of downsampling and upsampling blocks, each equipped with cross-attention layers that integrate the fMRI-derived conditioning. Since the conditioning input to SD) was parcel-wise embeddings, this can be represented as token matrix F. E TRPx/ (batch size e, E R corresponds the embedding of parcel P,. If anatomical or funetional labels are available for brain parcels, this formulation enables ROI-level probing of the cross-attention mechanism to see how brain representations interact with U-Net in the generative process. Following this idea, propose the Image- Brain BI -i-directi framework (IBBI) for exploring the internal attention dynamics, which links brain activity and image features during decoding.
1 for simplicity), where each row
4.1 PROBLEM SETUP
In NeuroAdapter, each cross-attention layer computes attention scores Attn(Q,, K, V), where queries Q E Raxd come from spatial tokens in the U-Net of SD, and keys and values (K, V) E Rxxd derived from the fMRI embeddings E. At each denoising timestep i, the attention weight matrix A (f.h.)) x for head h in layer l encodes the influence of each parcel token each spatial query. Each entry of the attention weight matrix can be expressed as:
exp (...C. K ((e.h.i)) /va) ΣE- exp (Q) (P.A.e) .K. k(l.b.tt) Vdd)
(EA..)) Aij
where query index i (1,..., 9) and parcel index j E (I,...,p).
p). Specifically, the entry A (8.h.0
refers to the attention from the i-th query vector Qf (f.h.4) to the j-th parcel token, represented by its key vector Ke (f.A.t). Intuitively, each entry of this matrix reflects the degree of attention from particular spatial query in the image to a specific parcel. Our proposed interpretability framework further exploits this matri from two complementary views.
5

---

Published as a conference paper at ICLR 2026
4.2 BRAIN-DIRECTED VIEW
We summarize the attention weight matrix ((,,h,)) over brain parcel tokens at each timestep into vector (C)) (parcel contribution vector), normalized to unit mass. Formally, let L be the number of cross-attention layers in U-Net, I be the number of multi-attention heads, and q' be the number of spatial queries in layer É, At the denoising step I, each cross-attention map satisfies Σ)=1 (f.h,4) for every (e, h, i). To aggregate the total attention mass assigned to each parcel across layers with different spatial resolutions, we weight every query equally and normalize by the total number of queries Σi= q². For each parcel j E {l....,P}. we define
I H E of
(.h..)) A
Here, Ej=1 B 1. so B(O) E RP can be interpreted as a query- guery-weighted share of attention mass over parcels at timestep L. The vector represents the relative contribution of different parcels.
4.3 IMAGE-DIRECTED VIEW
We are motivated by the previous work that interpreting text guidance in SD (Tang et al. (023b). In our case, the spatial structure A (t.h.t) enables us explore further where, in the generated image, each brain parcel or ROI (Region of Interest) directs its attention at timestep t. For a given ROI group from parcels, denoted as S (1.....P) .P). we pool attentions across heads and ROI tokens to form query-wise attention profile for each layer:
I H
(4.0 m R
(1)
= I
(E.N.)) An.
H R h=IjER
(E.() The vector m E R35 is then reshaped to a 2D map, which matches the spatial grid of the layer Because the spatial resolution varies across downsampling and upsampling blocks of the U-
Net, we upsample each 2D map to full image resolution, yielding U.(..) E TR. Hime x Wau for every cross-attention layer. To produce overlays that comparable for spatial location across ROIs, normalize each upsampled map to unit L1 mass and then average uniformly across layers:
U.C.0) (C,0)
( = L Z
Σ
ULL R
(x. v)
We refer ((() as the ROI attention maps, which highlights where a given ROI allocates its attention in the image at timestep Intuitively, ROI attention maps provide a functional footprint of each ROI the stimulus space, allowing us interpret the role of neural data from different parts of the brain in shaping specific image regions during reconstruction.
EXPERIMENTS
5.1 DATASETS
Natural Scene Dataset (NSD). We used the NSD, large-scale 7H-MMRI dataset designed for studying visual representations in the human brain (Allen at (022). This contains hishluesol brain responses from eight subjects, each viewing up to 10,000 distinct natural images sampled from the MSCOCO dataset (Lin et al.) 2014). In our experiments, we trained our brain decoding model and encoding model (see Section 3.4) the NSD data following the standard preprocessing steps. In the following sections, we report comparison with prior work using the averaged results from four subjects who completed all fMRI scanning sessions (subjects 2, 5, 7). For the relevant ablation studies, we restricted our analysis to subject and evaluated models under different experimental conditions this single-subject dataset.

---

Published as a conference paper at ICLR 2026
Ground Truth
Subject
Subject
iik
Subject
Subject 7
Figure 4: Ground truth with decoded stimuli from NeuroAdapter across 4 subjects.
NSD-eragery. We further evaluated framework the NSD-Imagery dataset (Kneeland al., 2025), an extension of the NSD designed to study brain activity during mental imagery. It contains high-resolution 7T-IMRI recordings from the same eight participants as NSD, with trials including simple geometric patterns, complex natural scenes, and conceptual word cues. During imagery runs, subjects were cued with letter and instructed vividly imagine the corresponding stimulus without physically seeing it. Each subject completed 12 runs (9 run types with imagery runs repeated twice), yielding 576 trials per participant. In evaluation, we directly tested our model, which was trained on NSD, on this dataset to see if our model can generalize to mental imagery tasks.
Deeprecon Dataset (Deeprecon). The Deeprecon dataset (Shen et al. 2019b) comprises fMRI activity data from five subjects who viewed both ImageNet images and artificial images. The dataset contains 1,200 distinct natural images for training (each presented with five repetitions), 50 natural images and 40 artificial images for testing (each presented over 20 repetitions), totaling 8,000 brain samples per subject. An important consideration for this dataset was that natural test images were selected from ImageNet categories that differed from the training categories, and artificial images were included as additional test stimuli. For this dataset, we trained our brain decoder and encoder on 16 brain parcels across two hemispheres, including early visual areas (V1, V2, V3), V4, higher-order visual regions (LOC, FFA, PPA), and the broader higher visual cortex (HVC) region.
5.2 EVALUATION METRICS
We evaluated the model's performance using the following eight image quality metrics that are commonly used in the literature. PixCorr measures the pixel-level correlation between reconstructed and ground-truth images. SSIM denotes the Structural Similarity Index Metric (Wang et al. 2004). AN(2) and AN(5) refer the 2-way classification (2WC) accuracy based features from layers and 5 of AlexNet (Krizhevsky et al. 2012)). respectively. CLIP corresponds to the 2WC accuracy of the output layer of the 4 Vision model (Radford al. [202T). Incep refers to the 2WC accuracy computed on the final pooling layer of Inception V3 (Szegedy et al. 2016). Eff and SWAV are distance-based metries computed using feature representations from EffiiietNeetB13 (Tan &
Le 2019) and SwAV-ResNet50 (Caron et al. 2020).
5.3 DECODING DYNAMICS ANALYSIS VIA CROSS ATTENTION
During inference, we applied 50 denoising steps for the reversed diffusion process, which a com- mon practice for Stable Diffusion (Appendix L. We extracted the full attention weight matrices

---

Published as a conference paper at ICLR 2026
A (f.h.t) across all layers E at each timestep. This yields step-by-step record of how brain repre- sentations influence different spatial queries throughout the generative trajectory. For grain-directed view, we computed a pareel contribution vector B(O)) showing the relative influence of each parcel at each timestep. We then projected this vector onto the cortical surface using pycortex (Gao al., 2015), visualizing how strongly each parcel influenced the generated stage. For image-directed view, mapped the spatial query tokens weighted by ROI-specific attention onto the pixel-level image grid, yielding heatmaps that highlight where each ROI attends. Then, we overlaid the ROI attention maps on NSD images for representative eategory-selectivy regions in human brain.
RESULTS
6.1 DECODING PERFORMANCE
We evaluated approach 8 image quality metrics (Section (.2). comparing it against prior single-subject decoding methods, including Cortex2Image (Gu et al. 2024). Takagi Nishimoto (2033) ; Brain Diffuser (Ozcelik & VanRullen 2023). MindEyee (Scotti al. 2023). and DREAM Xia et al. 2024). We further report results from recent multi-subject models, MindFormer (Han et al. 2024) and MindBridge (Wang et al., (0244)), which were trained using single-subject datasets for fair comparison. Also, we established a baseline model for each subject by retrieving an image from 1.3 million ImageNet images (Deng et al. 2009) whose predicted neural activity from our encoder best correlates with the ground truth fMRI response. following the spirit of earlier feature- matching based decoding approaches inspired by Kay et al. (2008). Examples of the baseline are shown in Appendix
B
From Fig. (a), we observe that NeuroAdapter achieves competitive performance with, and in some cases surpasses, anbeedding-aligned approaches on high-level semantic metrics. This pattern sug- gests that despite its simplicity, our model is particularly effective at capturing semantic content encoded in the fMRI signals without the use of an intermediate representation (Fig. 4 Appendix AA)
Additionally, our approach also captures low-level metrics reasonably well compared to the baseline retrieval method, although these improvements are more modest compared to those reported by other methods. To better understand it, we compared our performance with Brain Diffuseer models using different embedding spaces. As evident in Fig. (b), the better performance comes from the separate model pathway for predicting low-level latent features and removing them, as in the case of Brain-Diffuser w/o VDVAE, making their performance comparable By design, we chose not to include such a pathway in NeuroAdapter and instead have a more direct and interpretable link between brain activity and image reconstruction (see Section 6.2
on low-level metries.
We further compare how well brain activity predicted from the decoded images matches the mea- sured IMRI responses (Fig. 3 (b) in red). We also report the correlation between the predicted activity for the ground truth image and the corresponding measured fMRI responses (Fig. 3 (b) in blue). This figure shows that the decoded images have visual properties sufficient to elicit predicted neural activity similar to the activity evoked by original image, further strengthening our decoding results.
We report performance of our model on two additional datasets, NSD-Imager and Deeprecon, (Kneeland et al. 2025 Shen et al. 2019b) with quantitative and qualitative results reported in the Appendix Q and R On NSDD-mageed
NeuroAdapter demonstrates comparable generalization ability across both mental imagery and vision trials compared to existing work, especially for high- level semantic metrics. Experiments on Deeprecon, where training and test classes are disjoint, suggest that the model is able to infer not only category-level information but also finer low-level visual properties such shape (e.g., coin), orientation (e.g.. instrument), and color (e.g.. reddish reconstructions for pink artificial shapes). To our knowledge, no existing diffusion-based decoding pipelines have been quantitatively evaluated Deeprecon, and we provide our results baseline for future research.
6.2 DECODING INTERPRETABILITY
In this section, we visualize and analyze how brain representations influence the generative process with cross attention in NeuroAdapter. As mentioned in Section 4 our proposed IBBI framework
8

---

Published as a conference paper at ICLR 2026
Model Comparison
Ablation Study
160
Cortex2lnags
Brain Diffuser
Improvenent Over Baseline (50
140
Takagi al.
2059
Drain Diffier weo CLIP-Test Ted
Brein Diffieer
Brain-Difleer we CT CT.Po-ision
120
MindEyel
Brain-Dictuser we VOVA
100
DREAM
NeuroAdapter
MindFormer (6)
30
MindDridge
Improveneat
NeuroAdapter
20
PiaCoer
SSIM
ANO2
AN(S)
Incep
CLIP
SwAV
PixCorr SSTM AN(2) ANIS)
Incep CLIP EMT SwAV
Image Quality Metrics
Image Quality Metrics
Figure 5: Model Comparison. Decoding performance across eight image quality metrics, com- paring prior approaches and our method. To ensure fair comparison, results are shown as relative improvements over subject-specific ImageNet-retrieval baseline. (a) NeuroAdapter achieves com- petitive performance with umbedding-allgnee approaches, particularly on high-level semantic met- rics. (b) Comparison with Brain Diffuser variants shows that their advantage on low-level metrics arises from dedicated pathway for predicting latent visual features (VDVAE), whereas removing this pathway yields performance on low-level metrics comparable ours.
Target
Prediction
Brain Surface Plot
Target
Prediction
Brain Surface Plot
STOP
Figure 6: Example projections of averaged parcel contribution vector onto the cortical surface across denoising steps. Yellow colors denote parcels with strong influence across the denoising trajectory, while blue regions have a weaker contribution.
provides two complementary perspectives, showing how different brain regions contribute to visual reconstruction and where those ROIs direct their attention in the pixel-level stimulus space.
Based on the parcel contribution vector, we averaged (*) across timesteps to obtain global view B(A) summarizing parcel contributions throughout the generative process. The 200 parcels and their corresponding contribution weights were projected onto the cortical sur- face for visualization. For easy interpretation through the visualization (Fig. 6). ranked the parcels by their average contribution and divided them into five partitions (top 20%, 20-40% 60-60% 60-80%, and bottom 20%). This partitioning highlights the relative importance of dif- ferent cortical regions, enabling to identify high-impact parcels that dominate the generative trajectory and low-impact parcels that play only minor roles.
Brain-directed View.
Image-directed View. Here, we visualize the ROI attention maps (RAM) across generative timesteps for representative category-selective regions, including Face, Body, Scene, and Word. Fig. 7 reveals how different cortical ROIs guide attention toward distinct spatial locations in the image during the unfolding denoising process, thereby linking regional neural signals to specific pixel-level features. Additional examples of ROI attention maps are provided in Appendix NN
To further evaluate RAMs, computed Intersection-ove--miiee (loU) and Dice scores between ROI-specific IBBI masks and semantic segmentation masks from Segment Anything (SAM3; (Carion al. 2025)), which serve useudo-grount truth. For IBBI masks, each ROI produces

---

Published as a conference paper at ICLR 2026
2D attention map over denoising steps. We followed the approach from (Tang al. 2023b to obtain binary masks representing ROI-specific attended regions. A whole-image mask was used as an "attend everywhere" baseline. The quantitative results in Table of (rpendixO]showe that Face, Body, and Word ROIs have substantially higher loU and Dice scores with /BB/ masks compared to the whole-image baseline. Scene masks returned by SAM3 typically cover large, contiguous background regions, which inflates IoU/Dice for the whole-image baseline because most pixels belong to the "scene" class. Example segmentation maps are also included in Appendix O
Stop=1
Step=10
Step=20
Step=30
Step=40
Step=50
Predictions
Face RAM
Body RAM
Scene RAM
Word RAM
Figure 7: ROI attention map dynamics across generative timesteps. early denoising steps, when the image is still highly blurred, maps are broadly distributed; as the denoising progresses and structure emerges, the attention becomes selective, converging on regions relevant to the content.
Causal Perturbation Analysis. Having the parcel-wise linear mapping further allows us to per- form perturbation analysis, in which masked specific ROIs and examined how this manipulation altered the reconstructed images. Consistent with the selectivity of different ROIs, we observe that masking low-level ROIs does not compromise the semantic content of the generated images, but masking high-level ROIs completely changes them (See Appendix [P for ablation details).
7 DISCUSSION
We present an effective end-to-end brain-decoding framework that directly conditions the diffu- sion denoising process brain activity, bypassing intermediate feature spaces and enabling both effective decoding and mechanistic interpretability. Our results show that this approach achieves competitive reconstruction quality, particularly on high-level semantic metries. Due to the stochas- tic nature of the diffusion model, we observe large variability in the quality of the generated images. While our encoder based selection addresses this limitation some extent, future work will have to better understand the mapping from brain activity to images and make model performance more consistent. We believe this will be a great use case for interpretability methods in this domain.
Meanwhile, we notice that current brain brain- decoding benchmarks may be approaching saturation when evaluated solely through image quality metrics. Improvements these scores do not necessarily re- flect faithful brain decoding, as they may also result from stronger alignment with pretrained embed- ding spaces or simply the use of more powerful generative models. Therefore, our IBBI framework provides a complementary perspective, aiming to reveal how cortical parcels contribute to and shape the unfolding generative process, thereby linking brain activity and image features in a bi-directional manner. Looking ahead, future progress in brain decoding will depend both methodological ad- vances and richer interpretability frameworks, moving beyond metric-driven evaluation toward a deeper understanding of the neural-generative interface.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
Research reported in this publication was supported in part by the National Institute Neurological Disorders and Stroke of the National Institutes of Health under award numbers IRFINS128897 and AROINS128897 The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
REPRODUCIBILITY STATEMENT
We have made every effort ensure the reproducibility of our work. Details the datasets used, including NSD core, NSD imagery and Deeprecon, are provided in Section 5.1 The architecture of NeuroAdapter, training objectives and evaluation setup are described Section 3 Our inter- pretability framework (IBBI) is fully specified in Section including the mathematical definitions. We also provide results of ablation studies in appendices to verify the robustness our results. For computational reproducibility, L40 GPUs. Each model was trained for 300 epochs with a batch size of 16, requiring approximately 25 hours of training time. Source code, along with instructions for reproducing all experiments, is available at athttps::
models were trained
university GPU cluster with 2 NVIDIA
/ wtt.hab..com//rieeessottttt.t htt.ggsskoort-eabb//eurrddddappa
THE USE OF LARGE LANGUAGE MODELS (LLMS)
Large Language Models (LLMS) were used in this project as general-purpose assistant tools. Specif- ically, used GitHub Copilot with Claude 3.7 help sort and refactor code for readability and debugging during the research process, and used OpenAl ChatGPT-5 to polish the writing for clar- ity and effective communication of our ideas. No part of the model design, experimental results, or scientific conclusions depended on LLMs.
REFERENCES
Hossein Adeli, Sun Minni, and Nikolaus Kriegeskorte. Predicting brain activity using transformers. bioRxiv, pp. 2023-08, 2023.
Hossein Adeli, Minni Sun, and Nikolaus Kriegeskorte. Transformer brain encoders explain human high-level visual responses. arXiv preprint arXiv:2505./7329, 2025.
E. J. Allen al. A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelli- gence. Nature Neuroscience, 25:1 16-126, 2022. doi: 10. 10.10388/411593-02--000
Richard Antonello, Chandan Singh, Shailee Jain, Aliyah Hsu, Sihang Guo, Jianfeng Gao, Bin Yu, and Alexander Huth. Generative causal testing to bridge data-driven models and scientific theories in language neuroscience. arXiv preprint arXiv:2410. 12, 2024.
Vinamra Benara, Chandan Singh, John Morris, Richard Antonello, Ion Stoica, Alexander G Huth, and Jianfeng Gao. Crafting interpretable embeddings for language neuroscience by asking 1lms questions. Advances neural information processing systems, 37:124137, 2024.
Victoria Bosch, Daniel Anthes, Adrien Doerig, Sushrut Thorat, Peter König, and Tim Christian Ki- etzmann. Brain-language fusion enables interactive neural readout and in-silico experimentation, 2025. URL ptttps://arxiv>
org wwtp://w.or//bb///000000 23941
Marlène Careil, Yohann Benchetrit, and Jean-Rémi King. Dynadiff:: Single-stage decoding of im- ages from continuously evolving fmri. arXiv preprint arXiv:2505.14556 2025.
Nicolas Carion, Laura Gustafson, Yuan-Ting Hu, Shoubhik Debnath, Ronghang Hu, Didac Suris, Chaitanya Ryali, Kalyan Vasudev Alwala, Haitham Khedr, Andrew Huang, Jie Lei, Tengyu Ma, Baishan Guo, Arpit Kalla, Markus Marks, Joseph Greer, Meng Wang. Peize Sun, Roman Rädle, Triantafyllos Afouras, Effrosyni Mavroudi, Katherine Xu, Tsung- Mang--Han Wu, Yu Zhou, Lil- iane Momeni, Rishi Hazra, Shuangrui Ding, Sagar Vaze, Francois Porcher, Feng Li, Siyuan Li, Aishwarya Kamath, Ho Kei Cheng, Piotr Dollár, Nikhila Ravi, Kate Saenko, Pengchuan

---

Published as a conference paper at ICLR 2026
Zhang, and Christoph Feichtenhofer. Sam 3: Segment anything with concepts, 2025. URL https://arxiv.. http://www...///////11.11.1////.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS '20, Red Hook, NY, USA, 2020. Curran Associates Inc. ISBN 9781713829546.
Mathilde Caron, Hugo Touvron, Ishan Misra, Herv'e J' egoo,, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. 2021 IEEE/CVF International Conference Computer Vision (ICCV), pp. 9630-9640, 2021. URL https:
//api. www.smannniccccolaa.......
:
Zijiao Chen, Jiaxin Qing, Tiange Xiang, Wan Lin Yue, and Juan Helen Zhou. Seeing beyond the
brain: Conditional diffusion model with sparse masked modeling for vision decoding. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22710- 22720, 2023.
Fan I. Cheng, Tomoyasu Horikawa, Kei Majima, Misato Tanaka, Mohamed Abdelhack, Shuntaro C Aoki, Jin Hirano, and Yukiyasu Kamitani. Reconstructing visual illusory experiences from human brain activity. Science Advances, 9(46):cadj3906, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scall scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. leee, 2009.
Matteo Ferrante, Tommaso Boccato, Furkan Ozcelik, Rufin VanRullen, and Nicola Toschi. Through their eyes: multi-subject brain decoding with simple alignment techniques. Imaging Neuro- science, 2:1-21, 2024.
Matteo Ferrante, Tommaso Boccato, Grigorii Rashkov, and Nicola Toschi. Towards neural founda- tion models for vision: Aligning eeg, meg, and fmri representations for decoding, encoding, and modality conversion. Information Fusion, pp. 103650, 2025.
James S. Gao, Alexander G. Huth, Mark D. Lescroart, and Jack L. Gallant. Pycortex: an interactive surface visualizer for IMRL. Frontiers in Neuroinformatics, 9, September 2015, ISSN 1662-5196. doi: 10.3389/fming 001155500023 URL httt:://jounnn 10. / fninf. 2015.002200 00023/ abstract
Zixuan Gong, Qi Zhang, Guangyin Bao, Lei Zhu, Rongtao Xu, Ke Liu, Liang Hu, and Duoqian Miao. Mindtuner: Cross-subject visual decoding with visual fingerprint and semantic correction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 14247-14255, 2025.
Zijin Gu, Keith Jamison, Amy Kuceyeski, and Mert R. Sabuncu. Decoding natural image stimuli from fmni data with a surface-based convolutional network. In Ipek Oguz, Jack Noble, Xiaoxiao Li, Martin Styner, Christian Baumgartner, Mirabela Rusu, Tobias Heinmann, Despina Kontos, Bennett Landman, and Benoit Dawant (eds.), Medical Imaging with Deep Learning. volume 227 of Proceedings of Machine Learning Research, pp. 107-118. PMLR, 10-12 Jul 2024. URI. ing.eeddiing.......... ml prwss//ssssee.... v227/ gu24a html
Inhwa Han, Jaayeon Lee. and Jong Chul Ye. Mindformer: Semantic alignment of multi subte-s fmri for brain decoding. arXiv preprint arXiv: 2024.
Tiankai Hang, Shuyang Gu, Chen Li, Jianmin Bao, Dong Chen, Han Hu, Xin Geng, and Baining Guo. Efficient diffusion training via min-snr weighting strategy. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), pp. 7441-7451, October 2023.
T. Horikawa, M. Tamaki, Y. Miyawaki, and Y. Kamitani. Neural decoding of visual imagery during sleep. Science, 340(6132)6639-442 2013. doi: 10.1112//ccennee.12244 www. science.....ccccc..cc orgg / doi/ /bss/ 10. 1ee1e11//saass swwsecceeecceeeeeee.ee......ee.e 1234330
URL https::/
Tomoyasu Horikawa and Yukiyasu Kamitani. Attention modulates neural representation to render
reconstructions according to subjective appearance. Communications Biology, 5(1):34, 2022.
12

---

Published as a conference paper at ICLR 2026
Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3 2022.
Jingyang Huo, Yikai Wang, Yun Wang, Xuelin Qian, Chong Li, Yanwei Fu, and Jianfeng Feng- Neuropictor: Refining fmmi-to-image reconstruction via multi-individual pretraining and multi- level modulation. European Conference on Computer Vision, pp. 56-73. Springer, 2024.
Ethan Hwang, Hossein Adeli, Wenxuan Guo, Andrew Luo, and Nikolaus Kriegeskorte. In silico mapping of visual categorical selectivity across the whole brain, 2025. URL httpttt/s::/re org/abs/2510.222
Kendrick N. Kay, Thomas Naselaris, Ryan J. Prenger, and Jack Gallant. Identifying natural images from human brain activity. Nature, (52(7185)3352-333 March 2008. ISSN 0028-0836.
doi: 10. 10.1008//auue306113.
Reese Kneeland, Jordyn Ojeda, Ghislain St- Yve, and Thomas Nasclaris. Brain-optimized infer- ence improves reconstructions of fmri brain activity, 2023, UtLhttss://axxx wrg.ii.orggab.....
2312. 07705
Reese Kneeland, Paul S. Scotti, Ghislain St-Yves, Jesse Breedlove, Kendrick Kay, and Thomas Naselaris. Nsd-imagery: A benchmark dataset for extending fmri vision decoding methods to mental imagery. In 2025 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1188522288866 IEEE, June 2025. doi: 10. 20009///pp2233..2255...o 2025. 02687
Utthttp:
//dx. doi org 10. 1119/
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. Imagenet classification with deep con- volutional neural networks. In Proceedings of the 26th International Conference on Neural In- formation Processing Systems- Volume 1, NIPS 12, pp. 1097-1105, Red Hook, NY, USA, 2012. Curran Associates Inc.
Haoyu Li et al. Neuraldiffuser: Neuroscience-inspired diffusion guidance for fmri visual recon- struction. IEEE Transactions on Image Processing, 34:552-565, 2025. ISSN 1941-0042. doi: 10. 20110////ip2225555500555555
Sikun Lin, Thomas Sprague, and Ambuj K Singh. Mind reader: reconstructing complex images from brain activities. In Proceedings of the 36th International Conference Neural Informa- tion Processing Systems, NIPS '22, Red Hook, NY, USA, 2022. Curran Associates Inc. ISBN 9781713871088.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Pi- otr Dollár, and C. Lawrence Zitnick. Microsoft COCO: Common Objects in Context, pp. 740-755. Springer International Publishing, 2014. ISBN 9783319106021. 978-3-319 1009--000021111 1-0602-14488 .48. http::tt:///t doi http:///h........./o/....01///13
doi: 10.1007/
Andrew Luo, Maggie Henderson, Leila Wehbe, and Michael Tarr. Brain diffusion for visual explo- ration: Cortical discovery using large scale generative models. Advances in Neural Information Processing Systems, 36:75740-75781. 2023a.
Andrew F Luo, Margaret M Henderson, Michael J Tarr, and Leila Wehbe. Brainscuba: Fine-graine natural language captions of visual cortex selectivity. arXiv preprint arXiv:2310.04420. 2023b.
David Mayo, Christopher Wang, Asa Harbin, Abdulrahman Alabdulkareem, Albert Eaton Shaw, Boris Katz, and Andrei Barbu. Brainbits: How much of the brain are generative reconstruction methods using? In The Thirty-eighth Annual Conference on Neural Information Processing wwwppewwwwwe.tooortteee (ot.oruummii--KAAAAAAAAAAaY
Systems, 2024.
Shinji Nishimoto, An T Vu, Thomas Naselaris, Yuval Benjamini, Bin Yu, and Jack L Gallant. Re- constructing visual experiences from brain activity evoked by natural movies. Current biology, 144196664)1 2011.
13

---

Published as a conference paper at ICLR 2026
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, Russell Howes, Yao Huang. Hu Xu, Vasu Sharma, Shang- thang-Wen Li, Wojeiech Galuba. Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegou, Julien Mairal, Patrick Labatut, Ar- mand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
Furkan Ozcelik and Rufin VanRullen. Natural scene reconstruuttion from fmri signals using genera- tive latent diffusion, 2023, URL
05334
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. PmLR, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. Proceedings of the IEEE/CVF Con- ference on Computer Vision and Pattern Recognition (CVPR), pp. 10684-10695, June 2022.
Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-Net: Convolutional Networks for Biomedical Image Segmentation, pp. 234 241. Springer International Publishing, 2015. ISBN 9783319245744. doi:
URL http: www. wwg.o.o......ooooo
1007/978 319- 24574 28
Alexander Schaefer, Kong, Evan M Gordon, Timothy O Laumann, Xi-Nian Zuo, Avram Holmes, Simon B Eickhoff, and B T Thomas Yeo. Local-global parcellation of the human cere- bral cortex from intrinsic functional connectivity mri. Cerebral Cortex, 2809):3955-114, July 2017. ISSN 1460-2199. doi: 10.10933cercoo/bbxx177
Paul Scotti, Atmadeep Banerjee, Jimmie Goode, Stepan Shabalin, Alex Nguyen, Aidan Dempster, Nathalie Verlinde, Elad Yundler, David Weisberg, Kenneth Norman, et al. Reconstructing the mind's eye: fmri-to-image with contrastive learning and diffusion priors. Advances in Neural Information Processing Systems, (6:24705-24722 2023.
Katja Sceliger, Umut Güçlü, Luca Ambrogioni, Yagmur Güşlütüürk, and Marcel AJ Van Gerven. Generative adversarial networks for reconstructing natural images from brain activity. Neurolm- age, 181:775-785, 2018.
Guohua Shen, Kshitij Dwivedi, Kei Majima, Tomoyasu Horikawa, and Yukiyasu Kamitani. End- to-end deep image reconstruction from human brain activity. Frontiers in Computational Neu- roscience, 13, April 2019a. ISSN 1662-5188. doi: 00389//iccm.2219900222 URL http:
7/dx www.coo.opg org 10. 3389/ wwwp://noo....../coo//.../...... 2019.00021
Guohua Shen, Tomoyasu Horikawa, Kei Majima, and Yukiyasu Kamitani. Deep image recon- struction from human brain activity. PLOS Computational Biology. (51))e10066333 January 2019b. ISSN 1553-7358. doi: 10. 10.137//formma..chb 10066.0....3.. URL http: dxx doi org/
10. Jouraal pcbi 1006633
Ken Shirakawa, Yoshihiro Nagano, Misato Tanaka, Shuntaro C. Aoki, Yusuke Muraki, Kei Majima, and Yukiyasu Kamitani. Spurious reconstruction from brain activity. Neural Networks, 190: 107515, October 2025. ISSN 0893-6080. doi: 10. 10. w.l.e..eeet.2.5.10000. 7/dx. doi org/ O. 1016 neunet. 2025. 107515
URL http:
Oriane Siméoni, Huy Vo, Maximilian Seitzer, Federico Baldassarre, Maxime Oquab, Cijo Jose, Vasil Khalidov, Mare Szafraniec, Seungeun Yi, Michaël Ramamonjisoa, Francisco Massa, Daniel Haziza, Luca Wehrstedt, Jianyuan Wang, Timothée Darcet, Théo Moutakanni, Leonel Sentana, Claire Roberts, Andrea Vedaldi. Jamie Tolan, John Brandt, Camille Couprie. Julien Mairal, Hervé Jégou, Patrick Labatut, and Piotr Bojanowski. DINOv3, 2025. URL
abs/2508. 11.00.10110 10104
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pallern recognition, pp. 2818-2826, 2016.
14

---

Published as a conference paper at ICLR 2026
Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 14453-14463, June 2023.
Mingxing Tan and Quoc Le. EfficientNet: Rethinking model scaling for convolutional neural net- works. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th In- ternational Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pp. 6105-6114 6114. PMLR, 09-15 Jun 2019. URL https:://poceeeddgg/ .mlr.
press/ an 9a 9tm
Jerry Tang, Amanda LeBel, Shailee Jain, and Alexander G Huth. Semantic reconstruction of contin- uous language from non-invasive brain recordings. Nature Neuroscience, 26(5):858-88)) 2023a.
Raphael Tang, Linqing Liu, Akshat Pandey, Zhiying Jiang, Gefei Yang, Karun Kumar, Pontus Stenetorp, Jimmy Lin, and Ferhan Ture. What the DAAM: Interpreting stable diffusion using cross attention. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Proceedings of the 6lst Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.
Toronto, Canada, July 2023b. Association for Computational Linguis- URL. https: / accl http//aaacaaacccaacaa/caacccccaa logchhlogy.o org/2023.
tics. doi: 10.
ac wng. 3107
Shizun Wang, Songhua Liu, Zhenxiong Tan, and Xinchao Wang. Mindbridge: A cross-subject brain decoding framework. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1 11333-113 11342, 2024a.
Yanchen Wang, Adam Turnbull, Tiange Xiang, Yunlong Xu, Sa Zhou, Adnan Masoud, Shekoofeh Azizi, Feng Vankee Lin, and Ehsan Adeli. Decoding visual experience and mapping semantics through whole-braie analysis using fmri foundation models. arXiv preprint arXiv:2411.07121, 2024b.
Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transuctions on Image Processing, 13(4):600-612, 2004. doi: 10.1109/TIP.200.88 819861.
Weihao Xia, Raoul De Charette, Cengiz Oztireli, and Jing-Hao Xue. Dream: Visual decoding from reversing human visual system. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pp. 8226-8235, 2024.
Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. 2023.
Bohan Zeng. Shanglin Li, Xuhui Liu, Sicheng Gao, Xiaolong Jiang, Xu Tang, Yao Hu, Jianzhuang Liu, and Baochang Zhang. Controllable mind visual diffusion model. In Proceedings of the AAAI conference on artificial intelligence, volume 38, pp. 6935-6943, 2024.
15

---

Published as a conference paper at ICLR 2026
APPENDIX
A EXAMPLES OF DECODED STIMULI ON NSD
Ground Truth
Subject
Subject
IP
Subject
Subject 7
Ground Truth
STOP
Subject
STTOP
FTQ
Subject 2
STOP
Subject
Subject
STFO
Figure 8: Examples of ground truth with corresponding decoded stimuli across subjects
16

---

Published as a conference paper at ICLR 2026
B EXAMPLES OF IMAGENET RETRIEVAL BASELINES
We present shared retrieved images across 4 subjects in this figure. In our experiment, we created and evaluated baselines separately for each subject.
Ground Truth
Baseline
Ground Truth
Baseline
Ground Truth
STOP
TRUCKS de USE + RIGHT
Baseline
Figure Ground Truth vs. ImageNet Retrieval Baselines.
C SCHAEFER PARCELLATION
To represent brain activity at the regional level, we adopt the Schaefer cortical parcellation (Fig. 10 This provides functional subdivision of the cortex derived from large-scale resting-state fMRI. In our experiments, we compute vertex-wise wise Signal-to-Noise Ratio (SNR) and select top 100 parcels per hemisphere with the highest average SNR.
Figure 10: Top-100-SNF Parcels for each brain hemisphere displayed on the cortical surface.
17

---

Published as a conference paper at ICLR 2026
D MODEL PERFORMANCE ON NSD
Table 1: Performance across different image quality metrics
Method
Lew-Level
High-Level
PixCort f SSIM
1lex(2)
Alex(5)
Incep
CLIP
Eff 0
SwAV +
Baseline
.117
.242
80.98%
89.05%
74.62%
80.16%
.872
.518
Cortex2Image
.150
.325
.662
.665
Takagi et al.
83.0%
83.0%
76.0%
77.0%
Brain Diffuser
254
.356
94.2%
96.2%
87.2%
91.5%
.775
.423
MindEyel
.309
94.7%
97.8%
93.8%
94.1%
.367
DREAM
.274
.328
93.9%
96.7%
93.4%
94.1%
.645
.418
MindFormer (s)
.241
.352
93.5%
97.5%
93.5%
93.6%
.659
.356
MindBridge (s)
.148
.259
86.9%
95.3%
92.2%
94.3%
.713
.413
NeuroAdapter
.124
.307
84.54%
93.48%
90.79%
90.97%
.716
.408
Table 2: NeuroAdapter vs. Brain-Diffuser performance on data from Subject
Method
Low-Level
High-Levve
PixCorr
SSIM
Alex(2) Alex(5)
Incep f
CLIP
Eff
SwAV
Baseline
.125
245
82.88%
90.41%
76.98%
81.72%
.868
.517
BD
.305
.367
96.7%
97.4%
87.8%
92.5%
.768
.415
BD w/o CLIP-Text
.279
.333
95.6%
97.0%
87.9%
91.2%
.796
.436
BD w/o CLIP-Vision
.327
.433
93.9%
94.1%
84.7%
84.5%
.821
.509
BD w/o VDVAE
.143
.302
85.6%
93.0%
87.3%
92.6%
.775
.414
Adapter
.133
.296
86.22%
93.96%
92.15%
92.03%
.706
.396
E MODEL PERFORMANCE ON NSD-IMAGEE
Table 3: NSD-Imagery: Mental Imagery vs. Vision Trials
Method
Low-Level
High-Level
PixCort SSIM 1
1lex(2) Alex(5)
Incep f
CLIP 1
Eff
SWAV 1
NSD-Imagery Mental Imagery Trials
MindEyel
.086
.349
59.56
61.00%
52.03%
54.72%
948
.64
Brain Diffuser
.064
.401
52.14%
58.35%
52.73%
54.07%
.935
.585
iCNN
.108
.340
50.57%
55.25%
49.39%
41.72%
.994
.560
MindEye2
.036
.14
47.60%
55.38%
46.02%
50.78%
.966
.591
Takagi et al.
-006
.455
41.88%
40.19%
43.26%
40.08%
.976
.606
NeuroAdapter
.322
58.90%
58.71%
57.26%
60.04%
.970
NSD NSD--magery Vision Trials
MindEyel
218
.412
73.56%
80.81% 1%
62.44%
65.34%
.811
510
Brain Diffuser
.107
.455
60.34%
72.84%
60.95%
58.31%
.908
.555
iCNN
.224
.385
71.67%
81.35%
61.16%
49.03%
.926
.524
MindEye2
.161
.480
70.10%
77.52%
62.69%
65.93%
886
512
Takagi et al.
-013
.412
41.55%
39.26%
39.26%
43.01%
.969
.610
NeuroAdapter
.077
.342
75.76%
78.54%
68.18
70.45%
.945
.576
18

---

Published as a conference paper at ICLR 2026
F
MODEL PERFORMANCE ON DEEPRECON
Table 4: Performance on Deeprecon natural images
Condition:
Token dropout, num of preds
Low-Level
High-Level
PixCorr 1SSIM f
Alex Alex(5) Incep f
CLIP
Eff
SwAV 4
wo/ keep low-level,
.309
77.7%
86.6%
74.2%
81.0%
.552
keep low-level,
.310
79.1%
87.6%
74.8%
81.3%
.545
wo/ keep low-level, 16
.102
.314
80.0%
88.8%
74.6%
81.7%
.892
.546
keep low-level, 4
.088
.316
78.8%
86.9%
73.4%
80.8%
.908
.522
keep low-level,
.084
.314
79.5%
87.0%
72.6%
81.1%
.907
.550
keep low-level,
79.9%
87.0%
71.2%
80.0%
.908
Table 5: Performance on Deeprecon artificial shapes
Conditions
Token dropout, num of preds
Low-Level
High-Level
PixCorr 1SSIM f Alex(2)
Alex(5) IIcepp 1
CLIP t
Eff
SwAV
wo/ keep low-level,
.050
.844
63.0%
55.7%
51.2%
52.1%
.960
.622
wo/ keep low-level,
.485
63.2%
56.4%
53.2%
50.6%
.958
.622
wo/ keep low-level, 16
.067
.477
57.1%
53.4%
53.1%
52.4%
.961
.626
keep low-level,
.057
.470
59.7%
55.0%
51.3%
52.3%
.958
.622
keep low-level,
.056
.475
62.6%
54.7%
51.2%
52.8%
.958
.621
keep low-level, 16
.057
.478
61.8%
55.8%
52.9%
51.8%
.955
.622
G ABLATION STUDY: BRAIN TOKEN DROPOUT
We conducted ablation study to evaluate the effect of the proposed fMRI token dropout (TD)
strategy in training on decoding performance. As shown in Table 6 removing token dropout sub- stantially compromised performance across almost all metrics.
Table 6: Effect of parcel-wise token dropout (TD) model performance
Conditions
Low-Level
High-Level
PixComt
SSIM f
Alex(2)
Alex(5)
Incep f
CLIP
Eff +
SWAV
without TD
.038
.307
67.4%
75.5%
61.2%
64.8%
.974 .706
.666
with TD
.133
.2%6
86.22%
93.96 %
92.15% %
92.03 %
.3%6
H ABLATION STUDY: NUMBER OF CONDITION DIMENSION
Table 7: Effect of different condition dimension (CD) on model performance
Conditions
Low-Level
High-Level
PixCor f
SSIM f
Alex(2)
Alex(5)
Incep f
CL.I
Eff +
SWAV
1024
.116
.303
81.49%
91.64%
88.17%
90.33%
.432
CD = 960
.134
.302
85.49%
93.38%
91.32%
90.62%
.720
.400
CD = 768
.133
.296
86.22%
93.96%
92.15%
92.03%
.706
.396
CD 576
.132
.297
86.31 %
93.87%
90.42%
90.76%
.712
.400
CD 384
.136
.301
85.65%
94.16%
89.78%
90.13%
.725
.411
192
.122
.290
85.25%
94.17% %
91.33%
90.78%
.718
.417
19

---

Published as a conference paper at ICLR 2026
I ABLATION STUDY: NUMBER OF HIGHEST-SNR PARCELS
Table 8: Effect of number of highest-SNR parcels (p) on model performance
Conditions
Low-Level
High-Level
PixCort
SSIM f
Alex(2)
Alex(5)
Incep
CLIP 1
Eff 4
SwAV
p = 100
.106
.296
84.87%
93.65%
88.52%
90.41 %
.730
.414
p 200
.133
.296
86.22%
93.96%
92.15%
92.03% %
.706
.396
p 500
.094
.307
77.83%
88.93%
85.84%
88.66%
.767
.449
p 1000
.086
.297
76.37%
88.09%
83.95%
84.89%
.778
.457
J ABLATION STUDY: PARCEL-WISE LINEAR MAPPER
Table 9: Effect of parcel-wise linear mapper (LM) model performance
Conditions
Low-Level
High-Level
PixCort
SSIM 1
Alex(2)
Alex(5)
Incep
CLIP t
Eff 4
SWAV
with LM
.133
.296
86.22%
93.96% 82.47%
92.15% 76.20%
92.03%
.706
.396
w/o LM
.073
.388
73.81
78.11%
.843
.515
K ABLATION STUDY: BRAIN ENCODER AS RANKING TOOL
We further evaluate the role of the brain encoder as selection mechanism for decoded stimuli. Table I0 shows that increasing the number of candidate predictions consistently improves decoding performance. In addition, we conduct an additional experiment in which each test sample is decoded eight times with different random initializations. We report image-quality metrics in Table [] for three conditions: (i) the Highest-Corr candidate selected by the brain encoder, (ii) the Lowest-Corr candidate, and (iii) Random candidate drawn uniformly from the eight samples. The brain encoder consistently improves performance relative to Lowest-Cori and Random selections, but Random images occasionally score higher certain perceptual metrics, indicating that the encoder is not optimizing for image quality. Instead, it selects candidates that are most aligned with the neural data, highlighting role as neural-fidelity criterion rather than perceptual metric booster.
Table 10: Effect of encoder-based selection across different number of predictions
Conditions
Low-Level
High-Level
PixCorr
SSIM
Alex(2) Alex(5)
Incep 1
CLIP +
Etf
SWAV
num of preds .044
.292
79.0%
90.1%
89.5%
90.8%
.729
.417
num preds .105
of preds -2
.292
82.6%
91.7%
89.8%
88.7%
.733
.416
.293
84.0%
93.5%
90.1%
91.5%
.725
.408
num of preds =
.133
.296
86.22 %
93.96 %
92.15% %
92.03%
0.706
.396
Table 11: Results of encoder-based selection on predictions per test sample.
Conditions
Low-Level
High-Level
PixCorr f
SSIM t
Alex(2) 1
Alex(S) f
Incep
CLIP t
Eff 1
SWAV
Lowest Corr
0.109
0.286
80.88%
89.51%
89.81%
91.31%
0.720
0.404
Random
0.130
0.286
84.00%
91.30%
92.05 %
92.33%
0.708
0.395
Highest Cor
0.139
0.285
88.02%
93.87 %
91.91%
93.01 %
0.702
0.387
20

---

Published as a conference paper at ICLR 2026
L ABLATION STUDY: NUMBER OF DENOISING STEPS IN REVERSED DIFFUSION PROCESS
Regarding the number of inference steps, follow the default setting of 50 denoising steps used in Stable Diffusion for inference. Further, we evaluated decoding quality across a range of denoising steps (20-80). As shown Table 12 performance remains highly stable around 50 steps, and no monotonic improvement is observed with more steps. These results indicate that the number of diffusion steps is not a sensitive hyperparameter in our pipeline, consistent with prior observations in diffusion-basee brain decoding.
Table 12: Decoding performance across different numbers of denoising steps.
Steps
PixCorr f
SSIM
AN(2) f
AN(5) 1
Incep
CLIP
Eff 4
SWAV /
20
0.137
0.291
87.92%
95.46%
90.49%
90.34%
0.707
0.396
30
0.135
0.291
87.93%
94.50%
90.90%
90.93%
0.706
0.391
40
0.142
0.299
88.06 %
95.96%
91.58%
91.83%
0.705
0.388
50
0.143
0.289
87.96%
95.42%
91.72%
91.19%
0.705
0.389
60
0.141
0.288
87.26%
94.61%
91.50%
91.62%
0.702
0.386
70
0.137
0.284
87.40%
94.85%
91.46%
91.10%
0.700
0.389
80
0.138
0.285
87.13%
94.47%
90.77%
90.43%
0.704
0.389
M EXPLANATIONS OF MIN-SNR LOSS WEIGHTING
At each diffusion timestep i, the effective signal-to-noise ratio is defined as
SNR,
a
where O₁ denotes the cumulative product noise scheduling coefficients.
Without reweighting, high-SNR steps (early timesteps) tend to dominate the mean squared error (MSE) loss, while low-SNR steps (late timesteps) provide weaker gradients despite being more challenging and important for generation.
Ideally, the model should learn more from low-SNR noisy samples rather than overfitting to the easier, cleaner ones. Min-SNR weighting balances this trade-off by rescaling the per-timestep loss with
min (SNR, 7)
W
SNR,
where 7 is threshold hyperparameter (we set it 5.0 in training).
21

---

Published as a conference paper at ICLR 2026
N ROI ATTENTION MAP VISUALIZATION
To better interpret the ROI attention maps, we connect them to well-established functional regions. Because the Schaefer parcellation does not provide anatomical functional labels for individual parcels, we assigned labels by mapping parcels to the labels available in NSD. A parcel was assigned given label if more than 50% of vertices overlapped with that region. Using this mapping, we visualize the attention maps of the corresponding ROIs on generated images, tracking how their spatial influence evolves from noisy to clean across timesteps.
Stop=1
Step-10 10
St
S0
Step-40
Step-50
Predictions
Face RAM
Body RAM
Word RAM
STOP
STOP
Figure 11: ROI attention map dynamics across generative timesteps
Step-1
Step-10
Step-20
Step-30
Step-40
Step-50
Predictions
RAM
Body RAM
Scene RAM
Word RAM
Figure 12: ROI attention map dynamics across generative timesteps
22

---

Published as a conference paper at ICLR 2026
O EVALUATION OF IBBI ATTENTION MAPS
To quantify interpretability, we evaluate ROI-specific /BB/ attention maps using Intersection-over: Union (IoU) and Dice scores, which measure the spatial overlap between predicted attention regions and semantic segmentation masks from the latest Segment Anything (SAM3, (Carion et al. 2025 ) SAM3 provides high-quality region segmentation and serves as pseudo-grounde -round- -ruth for our gen- erated images. Among the 515 test reconstructions, 236 images contain valid semantic regions, including 38 Face, 195 Body. 27 Scene, and Word images.
For IBBI masks, each ROI produces 2D attention map over denoising steps. We average attention across steps, normalize, and apply 50% threshold obtain binary masks representing ROI-specific attended regions, following the DAAM procedure (Tang et al. (0236). As a baseline, we use whole-image mask, representing the trivial strategy "attending everywhere."
We compute Intersection-ove--nnioo (loU) and Dice between predicted masks and SAM3 masks. Face, Body, and Word ROIs show substantially higher loU and Dice scores with IBBI attention maps compared the whole-image baseline, demonstrating that IBBI reliably localizes semantically meaningful regions. Scene masks returned by SAM3 typically cover large, contiguous background regions, which inflates loU/Dice for the whole-image baseline because most pixels belong to the "scene" class. In contrast, IBBI allocates attention selectively to diagnostic subregions rather than spreading uniformly across the entire background.
Table 13: Evaluations IBBI attention masks and baseline compared to SAM3 segmenta- tions.
Method
IoU
Dice
Face
Body
Scene
Word
Face
Body
Scene
Word
Whole-image Mask
0.124
0.171 0.362
0.605
0.199 0.453
0.213 0.459
0.276 0.504
0.717 0.314
0.313 0.595
IBBI Attn Mask
0.322
0.198
Face
Body
Scene
Word
Pred
STOOPP
STOP
SAMB
STOOP
STOP
IBBI
STOOP
STOP
Figure 13: Quantitative evaluation of ROI attention maps using SAM3 segmentation.
23

---

Published as a conference paper at ICLR 2026
P CAUSAL PERTURBATION WITH BRAIN ROI MASKING
We present causal perturbation by masking out the related parcels of ROIs. Because the Schaefer parcellation does not provide anatomical functional labels for individual parcels, assigned labels by mapping parcels to the labels available in NSD. A parcel was assigned to a given label more than 50% of its vertices overlapped with that region. In 200 parcels with high SNR, 103 parcels were labeled for subject 1. Among them, 50 parcels were labeled as low-level ROIs. including VI, V2, V3, and V4, while 53 parcels were annotated as Face, Body, Scene and Word ROIs.
Table 14: Effect of ROI masking (high-level vs. low-level) on model performance
Conditions
Low-Level
High-Level
PixCorr f SSIM f
Alex(2)
Alex(5)
Incep
CLIP
Eff 4
SwAV +
No Masking
.133
.2%6
86.22%
93.96%
92.15% %
92.03% % 74.20% 50.37%
.706
.396
LL ROI Masking
.119
.290
66.13%
78.38%
73.15%
.891
.535
HL ROI Masking
.019
.289
55.42%
58.34%
50.70%
.911
.641
Tanget
ROI Masking
Low-level ROI Mesking
Highsleve R0I Masking
Faoe ROI Masking
Torget
ROI Masking
Low-level RDI Misking
Highelevd ROI Masking
Rody ROI Masking
Tanget
Masking
Lew-lavel
Moaking
Masking
Masking
Target
Masking
Lew-level ROI Misking
Masking
Worl Masking
STOP
STTOP
Figure 14: Visual Effect of brain masking on different ROIs
24

---

Published as a conference paper at ICLR 2026
Q EXAMPLES OF DECODED STIMULI ON NSD IMAGERY DATASET
Truth
Prediction
Ground Truth
Prediction
Ground Truth
banana
fruit
mammal
stripes
yellow
zebra
Prediction
Figure 15: Best decoded examples on NSD-Imagery mental imagery task
X
Ground Truth
Prediction
Ground Truth
Prediction
Figure 16: Best decoded examples on NSD-Imagery vision task
25

---

Published as a conference paper at ICLR 2026
Ground Truth
Prediction
Ground Truth
Prediction
Ground Truth
banana
fruit
mammal
stripes
yellow
zebra
STOP
Prediction
Figure 17: Worst decoded examples on NSD-Imagery mental imagery task
X
Ground Truth
Prediction
Ground Truth
Prediction
Figure 18: Worst decoded examples on NSD-Imagery vision task
26

---

Published as a conference paper at ICLR 2026
R
EXAMPLES OF DECODED STIMULI ON DEEPRECON DATASET
Ground Truth
Botts
Subject
Subject
Subject
Subject
Subject
Ground Truth
Subject
Subject
Subject
Subject
Subject
Figure 19: Decoded examples Deeprecon natural image dataset
27

---

Published as a conference paper at ICLR 2026
Ground Truth
Subject
Subject 2
Subject
Subject
Subject
-
Ground Truth
Subject
Subject
Subject
Subject
Subject
Figure 20: Decoded examples on Deeprecon artificial shape image dataset
28


Reviews:
[{"review_id": "xvZWafvbqd", "reviewer": "Reviewer_rWiu", "summary": "This paper introduces NeuroAdapter, an end-to-end framework for brain-to-image decoding that directly conditions a latent diffusion model on fMRI-derived parcel embeddings, bypassing intermediate image/text feature spaces. The key technical idea is to inject parcel-wise brain tokens into Stable Diffusion via an IP-Adapter–style cross-attention module while keeping the text encoder empty. To probe how brain signals guide generation, the authors propose IBBI framework, which analyzes cross-attention weights over diffusion steps from two complementary views: (i) a brain-directed view that quantifies parcel/ROI contributions over time, and (ii) an image-directed view that produces ROI attention maps highlighting where cortical regions attend in the generated image. Experiments on three datasets show that NeuroAdapter attains competitive reconstruction performance (especially on high-level semantic metrics) and provides intuitive interpretability visualizations.", "questions": "Please see the weakness section.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Contribution to interpretability is clear. Interpretability is important for both AI and neuroscience. The IBBI framework gives a principled way to link parcel-level signals to image-space attention at each diffusion step, which improves transparency of the decoding pipeline.\n\n- Interesting use of a brain encoder as a ranking tool. Using a separately trained brain encoder to score multiple decoded candidates by their predicted brain responses is a simple, effective mechanism to pick the final reconstruction.\n\n- Thorough experiments on three datasets. The paper evaluates on NSD (with multiple subjects), NSD-Imagery, and Deeprecon. The metric suite includes both low-level (e.g., PixCorr, SSIM) and high-level (e.g., AlexNet/CLIP/Inception-based) measures, with ablations on token dropout, parcel count, and condition dimension.\n\n- Comprehensive appendix. The supplementary materials provide substantial details (data processing, model choices, ablations, additional visualizations), which improve clarity and reproducibility.", "weaknesses": "- Interpretability claims rely primarily on qualitative evidence. Although Figures highlighting ROI contributions and attention maps are compelling, the paper does not rigorously quantify interpretability (e.g., ROI-level correspondence metrics, sensitivity/perturbation scores summarized across subjects/ROIs, or agreement with independent neuroscience priors/experts). To make the interpretability claim more convincing, the authors are suggested to report a quantitative metric for benchmarking interpretability specifically.\n- With proper metrics defined for interpretability, the authors are suggested to compare NeuroAdapter’s outputs directly to those from related attention-map–producing baselines to confirm the effectivess of the proposed approach on interpretability.\n\n- Compute/efficiency and model size are unclear. Because NeuroAdapter adds an IP-Adapter–style conditioning path and uses 50 denoising steps with candidate sampling before brain-encoder ranking, it is important to clarify whether inference time and parameter count remain comparable to competing methods. Given that the main tables/figures show NeuroAdapter is competitive on high-level metrics but not leading on low-level metrics, the method could be at a disadvantage if it is substantially larger or slower. \n\n- Moreover, it is unclear when reporting results in Figure 5 or Table 1, the brain encoder ranking tool was used or not. The method is interesting but pretty stand-alone to the NeuroAdapter itself. Other methods may benefit from the design too.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Iv3YOV2l9I", "reviewer": "Reviewer_SDwm", "summary": "The paper proposes NeuroAdapter, an end‑to‑end brain‑to‑image decoding framework that directly conditions a latent diffusion model on parcel‑wise fMRI tokens, bypassing intermediate latent feature spaces. It introduces IBBI (Image–Brain BI‑directional Interpretability) to analyze cross‑attention across diffusion timesteps, offering a brain‑directed view (parcel contributions) and an image‑directed view (ROI attention maps). Experiments on NSD, NSD‑Imagery, and Deeprecon show competitive performance on high‑level semantic metrics relative to embedding‑aligned baselines, along with extensive ablations. Code and training details are provided for reproducibility.", "questions": "- Beyond confirming known category selectivity, what new neuroscientific findings does IBBI enable? Can you formalize and test such hypotheses with statistics?\n\n- Given your use of a brain encoder and the standard role of encoding in neuroscience, please justify why decoding is preferable for the interpretability aims here—what questions can IBBI answer that a well‑designed encoding analysis could not?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- NeuroAdapter replaces U‑Net cross‑attention with an IP‑Adapter–style module and feeds only fMRI tokens. SD weights are frozen, and only the parcel‑wise linear mapper and new cross‑attention layers are trained. This separation makes the conditioning path easy to analyze and replicates the setup across datasets.\n\n- Results are reported on NSD, NSD‑Imagery, and Deeprecon, helping assess generalization beyond the main dataset. Qualitative grids and quantitative tables are provided in the main text and appendices.\n\n- The paper also analyzes cross‑attention dynamics. It reveals the contribution of attention at the pixel level throughout the generative process. It also uses parcel‑wise attention to quantify contributions at the image level. (That said, it is unclear whether these analyses yield any genuinely new insights beyond what was previously known. Please see weaknesses and questions.)\n\n- Ablations cover fMRI token dropout, condition dimension, # of high‑SNR parcels, etc.\n\n- The paper provides code, hardware/training schedules, and describes dataset preprocessing and metrics.", "weaknesses": "- While high‑level metrics are competitive, PixCorr/SSIM are below strong baselines (e.g., Brain Diffuser, MindEye1).\n\n- The ROI attention maps (Face/Body/Scene/Word) and parcel contribution maps mostly illustrate expected effects (e.g., category ROIs attend to corresponding image regions). The paper does not yet demonstrate novel neuroscientific insights uniquely enabled by IBBI.\n\n- If the emphasis is understanding brain–feature correspondences, a conventional encoding framework (predicting brain activity from controlled features) is arguably the more direct tool. The manuscript does not fully justify why decoding is the right or necessary vehicle for the questions posed. For example, consider how “face‑selective” cortex was originally identified: via encoding analyses that directly linked face features to measured brain responses. In the present paper, the ROIs used for interpretability (Face/Body/Scene/Word) are not discovered by their decoding models; they are imported via a label‑mapping heuristic, and the ROI‑attention visualizations and ROI‑masking largely reaffirm known selectivities rather than revealing new ones. This leaves unclear what scientific insight requires the decoding step (as opposed to a targeted encoding analysis).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kj4pidGtzr", "reviewer": "Reviewer_hBgN", "summary": "This paper introduces \"NeuroAdapter,\" a framework for reconstructing visual images directly from fMRI brain activity. It bypasses the common two-stage approach of mapping brain signals to an intermediate feature space, instead conditioning a latent diffusion model directly on fMRI representations. The goal is to achieve competitive image reconstruction quality while also improving interpretability. The authors also propose an \"Image-Brain BI-directional interpretability framework\" (IBBI) that analyzes cross-attention mechanisms to visualize how different brain regions contribute to the image generation process.", "questions": "- The methodology relies on a separate \"Brain Encoder\" to select the best reconstruction from a batch of generated candidates. How dependent are the reported \"competitive\" performance metrics on this selection process? What is the average quality of a randomly selected (or \"raw\") generated image from the model, and how does this compare to the \"best\" selected image?\n\n- The model maps the complex, high-dimensional fMRI activity from an entire brain parcel to a single embedding using a simple linear projection. Is this linear mapping truly sufficient to capture the rich information in that parcel, or does it serve as a significant information bottleneck? Were more complex, non-linear mappings (e.g., small MLPs per parcel) explored, and if so, how did they affect performance and training stability?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper proposes a framework that directly conditions a generative model on fMRI data, bypassing the intermediate feature mapping common in two-stage methods. A primary contribution is the IBBI interpretability framework, which is designed to analyze cross-attention weights to visualize how specific brain regions influence the image generation. The method also demonstrates competitive decoding performance, especially on semantic metrics, and shows generalization capabilities to mental imagery tasks and novel image categories.", "weaknesses": "- The model uses a \"Parcel-wise Linear Mapping\" to transform fMRI data from each brain parcel into an embedding. This is a strong simplification. It assumes that the complex, high-dimensional neural information within an entire parcel can be sufficiently captured by a single linear projection. This could limit the amount of neural detail the model can use.\n\n- The methodology section notes that for evaluation, multiple images are generated, and a separate \"Brain Encoder\" model is used to select the best one. This complicates the claim of an \"end-to-end\" framework, as the final, high-quality output relies on a post-processing selection step that is external to the generative model itself.\n\n- The paper itself notes \"modest\" performance on low-level metrics (like pixel correlation and SSIM). It justifies this by stating it intentionally omitted a separate pathway for low-level features (which other models use) to favor interpretability. While this is a reasonable design choice, the model sacrifices high-fidelity, pixel-accurate visual reconstruction for semantic coherence and interpretability, which is a key weakness.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Q3kP8g7gmj", "reviewer": "Reviewer_5Pr2", "summary": "This paper “Towards Interpretable Visual Decoding with Attention to Brain Representations” introduces NeuroAdapter, an end-to-end fMRI-to-image decoding framework that directly conditions a latent diffusion model on brain-derived tokens, bypassing intermediate image or text embedding spaces like CLIP. The approach integrates fMRI parcel-wise embeddings into Stable Diffusion via IP-Adapter–style cross-attention, allowing the model to learn mappings from cortical representations to visual stimuli. To interpret the generative process, the authors introduce the IBBI ramework, which analyzes cross-attention dynamics during denoising to reveal how specific cortical regions influence image formation. NeuroAdapter achieves nice results on NSD, NSD-Imagery, and DeepRecon datasets, and provides meaningful visualizations linking cortical regions to specific visual features. I think the work is a solid piece but lacks some comparison with recent work that could shadow his novelty.", "questions": "- The metrics used are not particularly informative given current saturation; however, the authors appropriately acknowledge this. The emphasis on qualitative and interpretability results is appreciated and well justified.\n\n- The examples are compelling, especially the attention visualizations and the demonstration of out-of-distribution (imagery) performance.\n\n- The use of the encoder as a ranker is an elegant and practical idea; expanding on its generalization capabilities or possible integration as a training-time guidance signal could strengthen the paper.\n\nThat said, I do have some questions for the authors.\n\n1) Could you elaborate on how the brain encoder–based ranking could be used during training (e.g., contrastive or self-distillation setups) rather than only at inference?\n\n2) Have you considered test-time augmentation (e.g., sampling multiple seeds or perturbations of brain tokens) to better generalize to imagery data? The input distribution shift could be a key factor here. The model is working to some extent but lacks the precision of the visual task.\n\n3) How does the interpretability of IBBI compare quantitatively to attribution methods such as SHAP or feature occlusion, maybe on yours or simpler models like Brain-Diffuser?\n\n4) Why parcel based approach instead of voxel-wise one? I'm curios I've seen both approaches working fine and would like to better understand what is in your opinion the advantage of this strategy.\n\n5) How do you expect NeuroAdapter to perform in a multi-subject joint training setting—does the architecture allow easy cross-subject fusion?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "Quite elegant approach! The model removes the intermediate embedding bottleneck and directly links fMRI data to the diffusion process, addressing a key interpretability limitation in current brain-to-image pipelines.\n\nThe IBBI framework offers a structured and quantitative way to visualize brain–image interactions via cross-attention.\n\nNeuroAdapter matches or exceeds existing diffusion-based decoders on several high-level metrics, even without CLIP conditioning. However it's not the best-in class (author don't claim that). I don't think this is really important given the technical contribution, interpretability gains and the fact the metrics are kind of satured.\n\nThe method performs reasonably well on imagery and DeepRecon datasets, suggesting decent cross-distribution generalization.\n\nOverall, the paper is well-organized, includes ablations, clear diagrams, and code availability.", "weaknesses": "While the paper is solid and promising, several important references and comparisons are missing or insufficiently discussed:\n1) Early encoder-based visual decoding: Nishimoto et al. (2011) pioneered similar end-to-end fMRI-to-video approaches, and this line of work should be acknowledged to contextualize the contribution, since Encoder here is used in a new fashion as a rescorer.\n\n2)Language decoding parallels: Huth and Antonello’s works on encoder-based and interpretable language decoding provide useful methodological analogies that could enrich the interpretability discussion.\n\n3)Interpretability precedents: Antonello et al. introduced early interpretability frameworks for language-to-brain decoding that align conceptually with IBBI’s aims.\n\n4) Cross-subject generalization: Ferrante et al.’s work on NSD cross-subject decoding is relevant for comparison and could be informative for assessing NeuroAdapter’s generalization strategy.\n\n5) Recent concurrent diffusion-based methods: The paper does not compare against or even mention Dynadiff (Meta, 2025), which jointly fine-tunes diffusion models with a brain module (akin to the IP-adapter here), achieving similar or superior results.\n\n6) Interpretability-focused diffusion pipelines: BrainDIVE and BrainScuba combine diffusion models and neural encoders for interpretability analysis, conceptually overlapping with interpretability findings here.\n\n7) Transformers as interpretability proxies: Kriegeskorte’s recent work on transformer-based brain encoders as interpretable models is directly relevant to the proposed analysis.\n\nI've been working in the field for years now and I perfectly understand that given the recent explosion of apporaches it's difficult to keep everything under track. Probably many other work could be cited here, but the ones I mentioned are particularly close to this work. Some of them are recent enough to be considered parallel concurrent work (for example a recently accepted paper at NeurIPS on interpretability on visual decoding with similar findings) but  I think integrating these citations and comparisons would better situate NeuroAdapter in the broader literature and clarify its novelty relative to ongoing work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)