Brain-IT: Image Reconstruction from fMRI via Brain-Interaction Transformer
Authors: Roman Beliy
Amit Zalcher
Jonathan Kogman
navve wasserman
michal Irani
Venue: ICLR 2026 Poster (2026)
Keywords: fMRI-to-Image Reconstruction
Brain Decoding
fMRI Decoding
Multiple Brains
Primary area: applications to neuroscience & cognitive science
TL;DR: Brain-IT is a brain-inspired approach for image reconstruction from fMRI trained on multiple brain; transfers to new brains with few data

Abstract: Reconstructing images seen by people from their fMRI brain recordings provides a non-invasive window into the human brain. Despite recent progress enabled by diffusion models, current methods often lack faithfulness to the actual seen images. We present ``Brain-IT'', a brain-inspired approach that addresses this challenge through a Brain Interaction Transformer (BIT), allowing effective interactions between clusters of functionally-similar brain-voxels. These functional-clusters are shared by all subjects, serving as building blocks for integrating information both within and across brains. All model components are shared by all clusters \& subjects, allowing efficient training with limited amount of data. To guide the image reconstruction, BIT predicts two complementary localized patch-level image features: (i) high-level semantic features which steer the diffusion model toward the correct semantic content of the image; and (ii) low-level  structural features which help to initialize the diffusion process with the correct coarse layout of the image. BIT's design enables direct flow of information from brain-voxel clusters to localized image features. Through these principles, our method achieves image reconstructions from fMRI that faithfully reconstruct the seen images, and surpass current SotA approaches both visually and by standard objective metrics.  Moreover, with only 1-hour of fMRI data from a new subject, we achieve results comparable to current methods trained on full 40-hour recordings.

Published as a conference paper at ICLR 2026
BRAIN-IT: IMAGE RECONSTRUCTION FROM FMRI VIA BRAIN- TRANSFORMER
Roman Beliy*
Amit Zalcher
Department of Computer Science Weizmann Institute of Science
Department of Computer Science Weizmann Institute of Science
Roman bel i y@we zmann ac. i1
Jonathan Kogman
Navve Wasserman
Department of Computer Science Weizmann Institute of Science
Department of Computer Science Weizmann Institute of Science
Michal Trani
Department of Computer Science Weizmann Institute of Science
ABSTRACT
Reconstructing images seen by people from their fMRI brain recordings provides non-invasive window into the human brain. Despite recent progress enabled by diffusion models, current methods often lack faithfulness to the actual seen im- ages. We present "Brain-IT". a brain-inspired approach that addresses this chal- lenge through a Brain Inseraction Transformer (BIT), allowing effective inter- actions between clusters of ffunciioalll-simiile bruin-voxels. These functional- clusters are shared by all subjects, serving as building blocks for integrating in- formation both within and across brains. All model components are shared by all clusters & subjects, allowing efficient training with a limited amount of data. To guide the image reconstruction, BIT predicts two complementary localized patch- level image features: (i) high-level semantic features which steer the diffusion model toward the correct semantic content the image; and (ii) low-level struc- tural features which help to initialize the diffusion process with the correct coarse layout the image. BIT's design enables direct flow of information from brain- voxel clusters to localized image features. Through these principles, our method achieves image reconstructions from fMRI that faithfully reconstruct the seen im- ages, and surpass current SotA approaches both visually and by standard objective metrics. Moreover, with only 1-hour of fMRI data from a new subject, we achieve results comparable current methods trained full 40-hour recordings. Project page can be found in: https://amitzalchrr.ppo githir.ttttbbhoi wtt.huubbb///rii-III
INTRODUCTION
Reconstructing visual experiences from brain activity (IMRI-to-image reconstruction) a key chal- lenge with broad implications for both neuroscience and brain-compute interfaces (Milekovic) 2018 Naci et al. 2012). Such reconstructions may provide a window into visual perception in the brain, enable the study of visual imagery (Cichy et al. 2012 Pearson et al. 2015). reveal dream content (Horikawa et al. 2013] Horikawa & Kamitani 2017). and even assist assessing disor- ders of consciousness (Monti al.] 20100] Owen al.) 2006). In typical image decoding setting, subjects view natural images while their brain activity is being recorded using functional Magnetic Resonance Imaging (fMRI). This produces paired data of images and their corresponding fMRI scans. The task is then to reconstruct the perceived image from new (test) fMRI signals.
"Equal contribution

---

Published as a conference paper at ICLR 2026
aa)
Reconstruction of seen images from fMRI using Brain-IT:
Seen Image
Seen Image
Seen Image
Seen Image
Scen Image
Recon
Recon
Recon
Recon
Recon
STOP
STOP
(b) Reconstruction with limited amount of training data:
Seen 15 min 30 min 1 hour 40 hours Image
15 min 30 min hour 40 hours Image
Seen
Figure 1: Reconstruction of seen images from fMRI using "Brain-IT". (a) Image reconstructions
using the full NSD dataset (40 hours per subject). (b) Efficient Tronsfer-leanine new subjects with very little data: Meaningful reconstructions are obtained with only 15 minutes fMR recordings. (Results on Subject 1)
Early work in this domain mapped fMRI signals to handcrafted image features (Kay et al. 2008
Nasclaris al. 2009 Nishimoto al. 2011). which were then used for image reconstruction. Sub- sequent studies employed deep learning methods (Beliy et al. 2019 Lin et al., 2019 Shen et al.) 2019). Significant progress in reconstruction quality was achieved through the introduction dif- fusion models (Chen et al., [2023] Ozcelik & VanRullen [2233] TTkagi & Nishimoto (002)). Recent works address the scarcity iMRI data by integrating information across multiple subjects (Scotti et al. 2024 Gong et al.] 2025 Liu et al., (025), though variability across individuals remains challenge (Haxby et al. 2011 Yamada et al.] 2015 Wasserman et al., 2024).
Despite these advances, significant performance gap in IMRI-based image reconstruction remains. SotA methods produce reconstructions which are visually pleasing. yet often unfaithful to the actual seen image. They deviate in structural aspects (e.g., position, color), and often also miss or distort some of the semantic content. These limitations stem from the reliance on generative priors in diffusion models, which can produce realistic images even with limited guidance from brain activity. We attribute this gap to the way representations are currently extracted from fMRI, their mapping into image features, and the incorporation of these features into generative models.
In this work, we introduce Brain-IT, a new approach for fMRI decoding, which produces recon- structions that more closely resemble the viewed images both structurally and semantically. At the core of our method is the "Brain Interaction Transformer" (BIT), model that transforms brain-voxel clusters into patch-level image features. BIT's design is inspired by principles of brain organization. Neural processing is distributed across many regions rather than centralized. In the visual cortex, spatial information is organized retinotopically, preserving the layout of the visual field. Different aspects of visual representation such as color, shape, and higher-level semantic features are localized to distinct yet interconnected regions (Kandel et al. 2013). Guided by these principles, BIT clusters functionally similar brain- voxels and 'summarizes each cluster with single Brain Token (vector). The resulting clusters (Brain Tokens) are shared across subjects, thus capturing similar roles across individuals, and support effective integration of information (both within and across brains). Brain Tokens interact to refine their representations and are mapped to localized image-feature tokens. This enables direct information flow from functional clusters to localized image features.
"brain- is
brain tissue, whose activity estimated via represented by a single scalar value.
2

---

Published as a conference paper at ICLR 2026
Our pipeline integrates two complementary branches, both driven by image features predicted by BIT: (i) A high-level semantic branch, whose predicted features steer the diffusion model toward the correct semantic content of the image; and (ii) A low-level structural branch, whose VGG-pree VGG-predicted features are inverted through a Deep Image Prior (DIP) framework to reconstruct coarse image layout. The low-level image establishes the global structure, while the diffusion process refines it under semantic conditioning. We adopt this design to combine diffusion's 's strong generative priors with VGG's structural fidelity and DIP's convolutional inductive biases. Combining BIT's novel localized feature predictions with the two branches yields reconstructions that preserve structural fidelity, semantic detail, and perceptual realism.
Brain-IT outperforms previous methods on all standard metrics, resulting in reconstructions that are more faithful to the seen images. Moreover, Brain-IT supports efficient transfer learning to new subjects with little subject-specific training data: with only I hour of fMRI data from new subject, it achieves performance comparable to existing methods trained on the full 40-hour dataset. This efficient transfer learning is enabled by our model's design: since the atomic training unit is a functional cluster of brain voxels shared across subjects, and all clusters use the same network weights, we are able to learn representations that generalize across individuals and can be adapted to new subjects with limited training data.
Our contributions are therefore as follows:
We present Brain-IT, a brain- insiinid approach for IMRI-to-image reconstruction that faithfully reconstructs seen images, yielding SotA results (both visually and by quantitative metrics).
We introduce a 'Brain Interaction Transformer' (BIT), which maps functional brain-voxel clusters localized image features, allowing effective integration of information across multiple brains.
A new approach to low-level image reconstruction from fMRI via Deep Image Prior (DIP), which accurately predicts the coarse image layout (forming powerful input to the diffusion process).
Transier-learning to new subjects with little data: Meaningful reconstructions obtained with only 15 minutes of IMRI recordings. Results on I hour are comparable to prior methods on 40 hours.
RELATED WORK
fMRI-to-image reconstruction is a well-established field, seeing significant progress in the past decade. Farly work mapped fMRI to handcrafted image features (Kay al., 2008 Naselaris al.,
2009 Nishimoto et al. 201T). followed by works that mapped fMRT to deep CNN features (Güçli & Van Gerven| 2015 Zhang et al. [0118a: Shen et al.) 2019). End-to-end methods have emerged
(Seeliger et al. 2018 St- Y vee & Naselaris Beliy et al. 2019), with later ones predicting latent
codes of VAES or GANs (Han et al. 2019 Lin al.; 2019) Mozafari et al. 2020 Qiao et al. 2020
Ren et al.] 2021). Recently. diffusion models have improved realism and faithfulness by turning
predicted features into high-quality images (Chen et al. 2023) Ozcelik & VanRullen 2023 Takagi
& Nishimoto 2023). In parallel, the community has increasingly focused on integrating information
across multiple subjects to improve generalization under limited data per subject (Scotti et al. 2024]
Gong et al. 2025 Huo et al.) 2024 Xia et al., 2024b| Ferrante et al. [2244] Liu et al. 2025 Shen
ct al. 2024). 4) Next, we review related efforts along 3 key axes: predicting image features from fMRI,
leveraging cross-subject information, and predicting intermediate low-level images.
Image Features Prediction. Current methods predict image features from fMRI, such as semantic
CLIP or VAE latent embeddings. Most use simple linear models or MLPs (Takagi & Nishimoto
Xia et al. 2024a; Wang et al., 2024b); others build on top with unCLIP diffusion priors (Gong et al. 2025| Scotti al. 2023). A key limitation of these methods is that they typically compress all voxels into a single global fMRI embedding via a fully connected layer before predicting image features. Since visual information is distributed across multiple distinct yet interconnected brain regions, such a projection fails to fully exploit this distributed nature of the brain. Two recent works introduce spatial voxel groupings (Huo al., 2024 (hhen et al. 2024). which use voxel patches in shared anatomical space. However, they predict a single global image representation, making it difficult reconstruct localized image information. Our approach addresses these issues by forming functionally shared voxel clusters which are mapped directly to localized image features. avoiding projection global fMRI embedding and yielding more accurate image-feature predictions.
2023
Cross-Subject Integration. Few prior works support multi--uubectt training (Lin et al., 2022 Fer-
rante et al. 2024 Gong al. 2025 Huo al. 2024 Scotti et al. 2024). A common aspect of these
3

---

Published as a conference paper at ICLR 2026
(a) Image Reconstruction Overview
IMRI signal
Semantic Features
Diffusion Model
Brain Interaction Transformer (BIT)
Low Level
VGG Features
V2C Mapping
Image Recon
(b)
Voxels to Clusters (V2C) Mapping
(c) Low Level Image Reconstruction via Deep Image Prior (DIP)
Brain-voxel clusters shared across all Subjects
Sub
(BIT) VGG Features
:
Noise
DIP Recon
MSE
(DIP) VGG
DIP
VGG
Sub N
Features
Figure 2: Overview of Brain-IT. (a) Brain Interaction Transformer (BIT) transforms fMRI signals into Semantic and VGG features using the Voxel-to-Cluse Cl-Csutter (V2C) mopping- Two branches are applied: (i) the Low-Level branch reconstructs a coarse image from VGG features, used to initialize the (ii) Semantic branch, which semantic features to guide the diffusion model. (b) Voxel-to-Cluster mapping (V2C): each voxel from every subject is mapped to functional cluster shared across subjects. (c) Low-level branch: VGG- predicted features are inverted using Deep Image Prior (DIP) reconstruct
image layout.
methods is that alignment is predominantly IMRI-centric: they treat each fMRI scan as single en- tity and rely on shared embeddings entire scans across subjects. As result, they can only exploit shared representations at the scan level, overlooking the more frequent similarities that exist between individual brain-voxels, both within a single subject and across different subjects. Inspired by the multi-subject Image-to-fMRI Encoder of (Beliy al. (024). we adopt voxel-centric model that shares network weights across all brain-voxels, both within and across subjects. By sharing most model components and leveraging voxel-level similarities, our approach integrates data effectively across subjects and adapts efficiently to new ones even with limited data.
Low-Level Image Reconstruction. Most existing methods regress directly from IMRI to the la- tent space of the diffusion model (e.g. VAE latent space) using MLPs or linear layers (Scotti al. 2024 Gong et al. 2025 Ozcelik VanRullen 2023). Alternatively. NeuroPictor (Huo et al.) 2024) manipulates internal feature maps within the U-Net of Stable Diffusion using features predicted from fMRI. Unlike these approaches, our method introduces a complementary low-level branch that predicts VGG-based representations inspired by LPIPS (Simonyan & Zisserman, [2014] [Zhang al., 2018b) and inverts them through Deep Image Prior (DIP) framework. This setup results in more faithful low-level predictions, reconstructing global structure and low-level visual features.
BRAIN-IT PIPELINE
Our Brain-IT pipeline reconstructs images seen by subjects directly from their fMRI activations. This section provides full explanation of the approach (Fig. (1), while the technical details and architecture of the BIT model are described in Sec. 4 The pipeline consists of two main stages: () Image Feature Prediction: The Brain Interaction Transformer (BIT) maps fMRI signals into meaningful visual features using a shared Voxel-to-Cluste mapping (Fig 2p). Two BIT models are trained: one predicts adapted CLIP embeddings (semantic features), and the other predicts VGG features (low-level). (ii) Image Reconstruction: The predicted features are used in two branches. VGG features are inverted through Deep Image Prior (DIP) framework to produce a coarse recon- struction (Fig. 2b. Semantic features condition diffusion model to generate image aligned with the fMRI signal. At inference, the DIP-based coarse reconstruction initializes the diffusion process, which is then refined under semantic guidance to produce detailed, faithful image.
3.1 IMAGE FEATURE PREDICTION
Given fMRI recording of subject viewing image, the goal is extract visual information by predicting image features that guide reconstruction. This stage has two main components: the Voxel-to- Coxel-to-Cllster Mapping, and the Brain Interaction Transformer.
4

---

Published as a conference paper at ICLR 2026
Seen Image
Bruin-IT (Ours)
MindEyc2 MindTumer NeuroVLA
Seen Image
Brain-IT (Ours)
MindEye2 MindTuner VLA
Figure 3: Comparing methods on 40-hour data (for Subject 1). Brain-IT is compared 3 leading methods, yielding reconstructions that better preserve both semantic content and low-level visual properties. Brain-IT better reconstructs the correct objects with relevant structural details (e.g. orientation, color), pro- viding reconstructions more faithful to the seen images. See many more examples Appendix Fig.
S2
functionally similar brain-voxels from all brains into few (128) shared clusters (Fig. 2j). Each cluster captures similar roles across all sub- jects/voxels, thus supporting effective integration of information both within and across brains. This Voxels-to-Clusterr ( (V2C) mapping assigns each voxel to a single functional cluster, reducing 40K voxels into 128 clusters. We determine the functional similarity of brain-voxels using the Encoding "voxel embeddings" obtained by the Brain Encoder of Beliy et al. (2024), which implicitly learns an individual embedding for each brain-voxel (for any subject). These encoded voxel embeddings capture the functional role of each brain-voxel, for example, whether it responds to low-level or semantic (high-level) visual features, whether it attends specifie spatial locations in image, or to the entire image globally, etc. To map voxels from all subjects into clusters, we apply a Gaussian Mixture Model (GMM) the voxel embeddings.
Toxels-to-Cluster Mapping (V2C). We start by mapping.uunctional
Brain Interaction Transformer (BIT). The BIT model takes as input the fMRI signals together with the V2C mapping and predicts image features (e.g., adapted CLIP embeddings for the seman- tic branch, VGG features for the low-level branch). First, BIT transforms brain activations into Brain Tokens, each summarizing the information from the voxels within voxel cluster. This pro- cess incorporates two learnable embeddings to integrate brain information: a per-voxel embedding, which captures each voxel's functionality, and per-cluster embedding, which captures each clus- ter's overall functionality. This serves as an information-selectinn voxel information is aggregated into Brain Tokens. The resulting Brain Tokens then refined via attention layers. Self-attention layers model relationships across Brain Tokens (voxel clusters), while cross-attention layers extract specific image feature information from them. This enables direct information flow from Brain Tokens to predict localized image features as demonstrated in Appendix Sec. C.4 by attention map visualization.
bottleneck by determining how
Enriching the Training Data. The training performed using pairs of [MRI measurements and the corresponding images viewed by the subject, taken from the NSD dataset (see Sec. 5.1 The model is trained on data from all subjects simultaneously. Since the subject-specific data is lim- ited, we further enrich the training set with "external images" (natural images without any fMRI). Specifically, we use ~ 10k natural images from the unlabeled portion of COCO dataset. To incor- porate these images, we employ the Image-to-fMRR Encoder of Beliy et al. (2024). which predicts IMRI responses for the external images, thereby creating additional training pairs. Such training is particularly important in Transfer-Learingg to new subjects, with very little subject-specific data.
3.2
IMAGE RECONSTRUCTION
Our image reconstruction consists of complementary branches: Semantic Image Generation that conditions diffusion model on adapted- CLIP embeddings to generate a semantic image, and Low- level Image Reconstruction that inverts VGG features via Deep Image Prior (DIP) to produce coarse image. At inference, we use Dual-Brand Generation: the low-level branch provides a coarse initialization, and the diffusion model refines it under semantic guidance.
5

---

Published as a conference paper at ICLR 2026
(a)
Brain Interaction Transformer (BIT)
Voxels activations
Brain Tokens
(c) Pross-Transformet
Image Features
module
Brain Tokenizer
Brain Tokens
Query Tokens
Cross Trasnformer
000
000
0000 JQ
0000
module
Query Tokens
V2C Mapping
K,V
Cross Attention block
(b) Brain Tokenizer
Voxels
Brain Tokens
activations
00
Self Attention block
K.V
Gross Attention block
Modulation
x5
K,V
Graph Attention
00000
Projection
Voxel
000
Cluster
Embeedingg
V2C Mapping
Embeddings
Image Features
0000
Figure 4: Architecture of the cntteractionTrrnsseen
(BIT). (Sec.
Semantic Image Generation. Our semantic branch (Fig. 2 a) aims to generate images which are simultaneously consistent with the distribution of natural images, and semantically faithful to the underlying seen image. It leverages the strong prior of diffusion models together with the adapted CLIP image embeddings predicted by BIT, to guide generation toward the seen image. We use the diffusion model introduced by (Scotti et al. (2024), which follows unCLIP-styLe approach: an adaptation of Stable Diffusion XL (SDXL) modified to condition on 256 spatial OpenCLIP ViT-bigG/14 tokens, enabling high-fidelity image reconstruction from CLIP tokens. Our training procedure consists two stages. (i) Feature Alignment: We first train the BIT model to predict the 256 spatial tokens of OpenCLIP ViT-bigG/14. This step aligns BIT's outputs with the conditioning signal expected by the diffusion model. Training uses paired fMRI activations and image CLIP embeddings with an 1.2 loss. (ii) Joint Training: In the second stage, we jointly train BIT and the diffusion model. BIT's predicted tokens are used to condition the image diffusion process. The two models are optimized together with the standard diffusion loss. Noise is added the target image, and the diffusion process is trained to denoise it. Importantly, the BIT output representation does need to remain aligned with the original CLIP embeddings. The two networks can establish representation that is better suited for conditioning image generation from fMRI activations.
Low level Image Reconstruction. Our low-level branch (Fig. c) aims to reconstruct the coarse structural layout and low-level visual features of the seen image, and is used to initialize the semantic-based diffusion process. Treating VGG features as a robust intermediate representation for low-level image reconstruction,
train BIT model to predict multi-layer VGG features ing the InfoNCE (van den Oord et al. 2018) loss, with each position in each layer predicted as separate token. At inference time, these predicted features are inverted through Deep Image Prior (DIP) (Ulyanov et al.) (2088) B) to generate a low level image. DIP provides a strong image prior through the inductive bias of convolutional neural networks. The DIP model is optimized to recon- struct an image whose VGG features are as close as possible the BIT-predicted VGG features (see (Fig. c).. To achieve this, DIP generated image is passed through a frozen VGG network, whose activations are optimized to match the BIT BlT-predicted features under L2 loss (details in Sec. D.2)
Dual-Branch Generation At inference time, the branches are combined: the low-level image initializes the diffusion process, providing coarse image structure, which is then refined into detailed reconstructions with semantic guidance. Reconstructions guided by semantic embeddings alone tend to be semantically consistent but not structurally faithful the visual stimulus, often failing to preserve low-level details of the seen image such as colors and structure. Kamb & Ganguli (2025) demonstrated that diffusion models generate images in a coarse-to-fine manner: first establishing global layout and then progressively refining details. Building on this insight, we initialize the diffusion process with a noisy version of the predicted low-level image, which supplies reliable structure. The diffusion model then updates this initialization under semantic conditioning, yielding reconstructions that capture both structure and semantic details, and are more faithful to the seen image. Finally, apply refinement stage in Scotti et al. (2024) in which the image is passed

---

Published as a conference paper at ICLR 2026
Low-Level
High-Levve
Method
PixCorr
SSIM
Alex(2) 1 A5ex(5) 1 Incep 1 CLIP Eff 4 SwAV
MindEye Scotti et al. 2023
0.319
0.360
92.8%
96.9%
94.6% 93.3% 0.648
0.377
Brain- Diffuser Oonelli & VonRaallon 2023
0.273
0.365
94.4%
96.6%
91.3% 90.9% 0.728
0.421
Takagi & Nishimoto
2023
0.246
0.410
78.9%
85.6%
83.8%
82.1% 0.811
0.504
DREAM al 2024a 20244
0.274
0.328
93.9%
96.7%
93.4%
94.1% 0.645
0.418
UMBRAE
20244b
0.283
0.341
95.5%
97.0%
91.7%
93.5% 0.700 0.393
Neuro
Shon ol.
2024
0.265
0.357
93.1%
97.1%
96.8%
97.5% 0.633
0.321
MindBridge
ge
Wung ot al.
20246
0.151
0.263
87.7%
95.5%
92.4%
94.7% 0.712 0.418
NeuroPictor
of
2024
0.229
0.375
96.5%
98.4%
94.5%
93.3% 0.639
0.350
MindEye2
Scotti et al.
2024
0.322
0.431
96.1%
98.6%
95.4%
93.0% 0.619
0.344
MindTuner
Goog et
2025
0.322
0.421
95.8%
98.8%
95.6%
93.8% 0.612
0.340
BrainIT (Ours)
0.386
0.486
98.4%
99.5%
97.3%
96.4% 0.564
0.320
Results on I hour (out of 40)
MindEye2 (1 hour)
0.195
0.419
84.2%
90.6%
81.2%
79.2%
0.810
0.468
MindTuner (1 hour)
0.224
0.420
87.8%
93.6%
84.8%
83.5%
0.780
0.440
BrainIT (1 hour)
0.331
0.473
97.1%
98.6%
94.4%
93.0% %
0.648
0.370
Table 1: Evaluation of different methods. We report low- and high-level metrics comparing Brain-IT with other reconstruction methods. Top: methods trained on the full 40 hours data. Bottom: results with only I from hour of subject-specifie data, All results are averaged across Subjects 1.2.5.7 from NSD. Brain-IT outperforms all baselines in 7 of 8 metrics, demonstrating strong semantic fidelity and structural accuracy. Importantly with just I hour of data, Brain-IT comparable to prior methods trained on the full 40 hours.
through a pre-trained SDXL diffusion model without any text prompt, producing visually enhanced reconstructions (see Sec. A.4).
4 BRAIN INTERACTION TRANSFORMER (BIT)
The BIT model predicts image features from voxel activations (fMRI) (see Fig. 4a). The Brain Tokenizer maps the fMRI activations into Brain Tokens, which are representations of the aggregated information from all the voxels of a single cluster (one token per cluster). The Cross- Transformer Module integrates information from the Brain Tokens to refine their representation, and employs query tokens to retrieve information from the Brain Tokens and transform it into image features, with each query token predicting single output image feature.
Brain Tokenizer: The Brain Tokenizer (Fig. 4b) transforms fMRI brain activations into infor- mative Brain Tokens, where each Brain-Token "summarizes' the information of a single func- tional cluster of boxin-voxels. The architecture of the Tokenizer detailed in Fig. 4 is explained next. It incorporates two learnable embeddings, initialized randomly and optimized during training: () Voxel Embedding: a 512-dimensional vector, similar to Beliy et al. (2024). trained capture each voxel's functionality. Importantly, while the voxel embedding of Beliy et al. (2024) were optimized for image-to-fMel fMRI Encoding, our voxels embeddings are optimized for fMRI-to-Image Decoding (which results in very different voxel embeddings). Each voxel embedding (vector) mod- ulates the fMRI activation (scalar) by multiplying the voxel activation with its corresponding Voxel Embedding. (ii) Cluster Embedding: a
vector for each cluster of voxels, trained bottleneck by
to capture the cluster's ' overall functionality. This serves as an information-selccttiii determining how voxel information is aggregated, and is shared by all subjects. The Brain- Tokenizer forms Brain Tokens by combining modulated voxel activations (voxel activations multiplied by their voxel embeddings) within each functional cluster. This is implemented with single-head graph attention layer (Shi et al. 2021). which restricts attention to act between each voxel and its corre- sponding Cluster Embedding according the -ecuuttt (V2C) mapping. The modulated voxel activations are used as keys (K) and values (V), while the learned Cluster Embeddings act as queries (Q). The output a set of 12-dimensional Brain Tokens, each representing one cluster.
Cross- Transformer: The Cross- Transformer Module (see Fig. c) refines the representation of Brain Tokens through their interactions. It integrates their information via query tokens to predict image features, with each query token corresponding single output image feature (architecture
7

---

Published as a conference paper at ICLR 2026
Seen Image
Bruin-IT (Ours)
Seen Image
Brain-IT (Ours)
MindTTuner MindEye2
MindTTuner MindEye2
Figure 5: Reconstruction with limited amount of subject-specific data (1 hour). We compare Brain-IT against 2 leading approaches which provide also I-hour reconstructions (MindEye2 & MindTuner) for Subjl. Brain-IT demonstrates greater fidelity to the seen image. See many more examples in App.Fig 2 S3
adapted from (Fu et al. 2.24)). ). Self-attention layers model interactions across voxel clusters, while cross-attention layers enable direct information flow from Brain Tokens to predict localized image features. In the first cross-attention block, query tokens (initialized randomly and optimized during training) attend to the Brain-Tokens, which serve as keys (K) and values (V). This is followed by five cross-transien transformer blocks, each consisting of a self-attention layer that refines the Token representations, and cross cross-attention layer that refines the predicted image features. Both self- and cross-attention layers contain eight heads. A final projection layer after the last cross-attention block maps the representation to the dimensionality of the output image features.
EXPERIMENTAL RESULTS
This section presents the effectiveness of Brain-IT for reconstructing images from brain signals (fMRI). We first briefly describing the experimental framework S... We then show quantita- tive and qualitative results from our full pipeline, showing that it outperforms current SotA methods 5.2). Next, we evaluate the transfer learning capabilities of Brain-IT, showing that training on new subject with only hour of data produces high-quality results, comparable to previous methods trained the full 40 hours, and significantly better than ones trained on I hour (Sec. (.3). Additionally, we demonstrate the robustness of our model through qualitative evaluation of image reconstructions "NSD Synthetic" Gifford al. (0024). which is dut-of-distribuutin dataset (Fig. S77)
5.1
EXPERIMENTAL. SETTING
Dataset. We used the Natural Scenes Dataset (NSD) (Allen et al. 2022 a large publicly available 7-Tesla fMRI dataset that records fMRI of subjects they viewed diverse images drawn from COCO (Lin et al. 2014). The dataset contains ~~73,000 Image-IMRI pairs, comprising ~99,000 unique images per subject and 1,000 images shared by all subjects. As prior NSD studies, we used the shared images as the test-set. For voxel selection, we adopted the eet-processed version provided by Gifford et al. (2023), which includes 440k voxels, mainly vision-related cortical areas.
Evaluation Metries. We follow the standard practice of evaluating model performance using both low- and high-level image metrics. Unless stated otherwise, our tables display averaged results across four subjects from the NSD dataset (Subjects 1,2,5,7), be compatible with previous meth- ods. PixCorr denotes the pixel-wise correlation between ground-truth and reconstructed images, while SSIM refers to the (grayscale) structural similarity index (Wang et al. 2004) Alex(2), Alex(5), Incep, CL.PP, measure two-way identification accuracy based on correlations between image and brain embeddings at different feature levels. Finally, EfficientNet-BI ("Eff") (Tan 2019) and SWAV- ResNet50 (" ("SwAV") (Caron et al.) 2020) report average correlation distance, where lower values indicate better performance. Moreover, in Table T6 we evaluate our reconstructions on 3 additional metrics: 1000-way CLIP retrieval, LPIPS (Zhang al. 2018b). and Color-SSIM.
5.2 RECONSTRUCTION RESULTS
We compare both quantitative and qualitative image reconstructions results to previous prominent methods. In Table I report both low-level and high-level quantiittvve evaluation metries. Across
8

---

Published as a conference paper at ICLR 2026
level brunch
Seen Image
brauch
Branches
Low-Level Metries
High-Level Metries
Method
SSIM f Alex(5) f
Incep
CLIP f
Low-level branch
0,505
99.1%
95.9%
85.8%
Semantic branch
0.431
99.4%
96.1%
95.2%
Combined
0.486
99.5%
97.3%
96.4% %
Figure 6: Two-Branch Contribution Re- constructions, Sample results of the semantic branch, low-level branch, and both combined.
Table 2: Branch contributions. Quantitative results for each reconstruction branch alone (semantic & low- level), and both combined.
all 4 low-level metrics, Brain-IT surpasses all previous approaches by a large margin, We attribute this the combination of our high-quality BIT predictions of low-level VGG features, along with their DIP inversion pipeline. For high-level metrics, our method achieves the best performance in 3 metrics and ranks second one, where NeuroVLA performs better. Overall, our method demonstrates strong performance across both high-level semantic and low-level structural fidelity.
Fig. provides visual comparisons of our method three prominent approaches. Many more exam- ples and comparisons to additional methods can be found in Fig. [S2 and reconstructions from other subjects are shown Fig.(S4) SS4] Brain-IT achieves significantly higher fidelity to the seen image than prior methods, both semantically and structurally (e.g., the correct objects, their spatial arrange- ments, color and shape). Despite the inherent resolution limits fMRI, our model successfully reconstructs most of the relevant features of the original images.
5.3 TRANSFER LEARNING
Brain-IT facilitates highly efficient transfer learning to subjects with very little subject-specific data. This is particularly important since collecting fMRI recordings is costly and timecconsuming All model components shared by all subjects, with the exception of the voxel embeddings which are subject-specific. These per-voxel vectors (which are part of the Brain Tokenizer), are optimized to capture the functionality of each voxel and to integrate information for predicting image features from this voxel's 's iMRI activation. Adapting to a new subject therefore requires optimizing only the voxel embeddings, whose expressivity is constrained by the frozen network components. This enables efficient adaptation even with very little training data. For details, see Sec. D.4
Table [ (bottom) shows quantitative comparison of Brain-IT against MindEye2 and MindTuner, all trained with only hour of data from a new subject. Brain-IT performs significantly better on all metries. Note that our model shows larger performance gap over competitors with hour of data than with 40 hours. Importantly, with just 1 hour of data, our results are comparable to previous methods trained on the full 40-hour dataset, highlighting the strong efficiency of our approach. These findings are further visually illustrated in Fig. where our 1-hour reconstructions surpass MindEye2 and MindTTuner trained on the same duration. Many more examples, as well as direct visual comparison between --hour and 40-hours reconstructions, are provided in Fig. S3 We further demonstrate that our method maintains strong performance with only 30 minutes or even 15 minutes of data. Visual examples are shown Fig. [ and Fig. S3 with quantitative results provided in Table T7 To our best knowledge, this is the first demonstration of high-quality reconstructions from as little as 15 minutes of fMRI data, demonstrating the generalization power of our method.
ABLATIONS AND ANALYSIS
This section evaluates the contribution of the two branches (semantic and low-level) and provides additional analysis. Further ablations of other components and factors are provided Sec. A Those include contribution of: (i) External data, incorporating unlabeled images with no fMRI into train- ing (Sec.[A.T): (ii) Functional vs. Anatomical clustering, comparing voxel clusters based on func- tional representations (as used in Brain-IT) vs. anatomical based clustering (Sec. (Sec.A.2) (iii) Number of clusters, examining the effect of the number of clusters on reconstruction performance (Sec. A.3).. We also provide interpretability analysis of Brain-IT via BIT cross-aties attention maps, showing spa- tial and semantic selectivity of brain tokens.
Two-Branch Contribution:
Fig. Fig. Gjshwss sample reconstructions using the semantic branch alone, the lew-level branch alone (via DIP without diffusion), and both branches together (more examples

---

Published as a conference paper at ICLR 2026
in Fig. 58 The semantic branch captures the semantic content of the image, while the low-level branch preserves coarse outlines and structural detail. The low-level branch provides strong struc- tural cues: in the computer image it captures the colors of the screen and desk, and in the giraffe image recovers the outlines of the two giraffes. When combined, the two branches yield re- constructions that more semantically accurate and structurally faithful. "able reports selected quantitative metrics (full results in T4) consistent with these observations: the low-level branch achieves higher scores on structural fidelity metries such as pixel correlation and SSIM, while the semantic branch excels on perceptual metrics. Notably, the combined reconstructions outperform both individual branches on several metrics, confirming their complementary contributions.
Brain-token selectivity. To better understand how Brain-IT uses fMRI information, analyze the BIT cross-attention maps between brain tokens (voxel clusters) and query tokens (image-feature positions). Averaging attention across layers and heads reveals that different brain tokens consis- tently contribute to specific spatial locations in the predicted feature maps, exhibiting a clear con- tralateral organization. Beyond spatial effects, some tokens preferentially attend to regions associ- ated with particular semantic concepts (e.g., faces, limbs, tex(), indicating semantic selectivity. Full visualizations are provided in Appendix C.4 (Figs. S5 and S6
DISCUSSION
We introduce Brain-IT, brain-inspired framework for reconstructing images from fMRL. It enables direct flow of information from finctional clusters of brain-voxels, to localized image features (both low-level structural image features, and high-level semantic ones). The shared functional clusters allow effective integration of information across multiple brains, and efficient Transfer-Learning new brains, yielding high-quality reconstructions with very little subject-specific training data. Brain-IT overcomes key limitations of prior approaches, providing high-quality fMRI-to-Image reconstruction, with significantly better structural and semantic similarity to the real seen image. Despite these advances, reconstructions remain imperfect, with semantics and fine-graines details sometimes inaccurate (see Appendix Fig. S9 for failure examples). While this may reflect limi- expressive feature spaces where
tations of the fMRI signal itself, future work could explore current method fails. On a different frontier - our Brain-Trann Trans--raanrrmmr (BIT), which enables direct interactions between different functional brain sub-regions, may be applied to other neuroscience ap- plications. By analyzing the information flow between functional sub-regions, the model may help reveal what is represented where in the brain, how inter-regional interactions support visual process- ing, and how these relate to perception. Our method's strong Transfer-Learnin capabilities may further make it suitable for more diverse neural signals and datasets, allowing pretrained model to be adapted to new studies with minimal data to answer task task-specific neuroscientific questions.
8 ACKNOWLEDGMENTS
This research was funded by the European Union (ERC grant No. 101142115).
10

---

Published as a conference paper at ICLR 2026
1. ETHICS STATEMENT
We use only publicly available datasets where all the necessary measures to ensure ethical conduct were made. Additionally this work is a continuation of line of works on brain decoding and is natural extension of previous works.
II. REPRODUCIBILITY STATEMENT
We provide the details of our method in the main paper and additional technical information in Sec. D Upon publication, we will release our full code with saved checkpoints to ensure that all results are easily reproducible. We will also share all reconstructed images, enabling researchers to easily compare our method under new evaluation metrics.
REFERENCES
Emily J Allen, Ghislain St-Yves, Yihan Wu, Jesse Breedlove, Jacob Prince, Logan Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, et al. A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence. Nature neuroscience, 25(1): (10:::16-1110 16-166,,6 2022.
Roman Beliy, Guy Gaziv, Assaf Hoogi, Francesca Strappini, Tal Golan, and Michal Irani. From voxels to pixels and back: Self-supervision in natural-image reconstruction from fmri. Advances in Neural Information Processing Systems, 22:6514-6524, 2019.
Roman Beliy, Navve Wasserman, Amit Zalcher, and Michal Irani. The wisdom of a crowd of brains: A universal brain encoder. arXiv preprint arXiv:2406. /2/79, 2024.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Un- supervised learning of visual features by contrasting cluster assignments. CoRR, abs/2006.09882, 2020. URL https: //apxvv.//aiivaa/aa org abs abw.//b..//.2.... 09882)
Zijiao Chen, Jiaxin Qing, Tiange Xiang, Wan Lin Yue, and Juan Helen Zhou Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding. In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22710- 22720, 2023.
Radoslaw M Cichy, Jakob Heinzle, and John-Dylan Haynes. Imagery and perception share cortical representations of content and location. Cerebral Cortex, 22(2):372-380 2012.
Matteo Ferrante, Tommaso Boccato, Furkan Ozcelik, Rufin VanRullen, and Nicola Toschi. Through their eyes: multi-subject brain decoding with simple alignment techniques. Imaging Neuro- science, 2:1-21, 2024.
Letian Fu, Long Lian, Renhao Wang, Baifeng Shi, Xudong Wang, Adam Yala, Trevor Darrell, Alexei A. Efros, and Ken Goldberg. Rethinking patch dependence for masked autoencoders. arXiv preprint arXiv:2401. 14391, 2024.
Alessandro T Gifford, Benjamin Lahner, Sari Saba-Sadiya, Martina G Vilas, Alex Lascelles, Aude Oliva, Kendrick Kay, Gemma Roig, and Radoslaw M Cichy. The algonauts project 2023 chal- lenge: How the human brain makes sense of natural scenes. arXiv preprint arXiv:2301.03/98, 2023.
Alessandro T Gifford, Radoslaw M Cichy, Thomas Naselaris, and Kendrick Kay. A 7t fmri dataset of synthetic images for -iit-ri--itittiionn modeling of vision. bioRxiv, 2024. doi: 10.1201/ 08. 18.64.604118.
Zixuan Gong, Qi Zhang, Guangyin Bao, Lei Zhu, Rongtao Xu, Ke Liu, Liang Hu, and Duoqian Miao. Mindtuner: Cross-subject visual decoding with visual fingerprint and semantic correction. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 14247-14255, 2025.
11

---

Published as a conference paper at ICLR 2026
Umut Güglü and Marcel AJ Van Gerven. Deep neural networks reveal gradient in the complexity of neural representations across the ventral stream. Journal of Neuroscience, 15027):10005100014 2015.
Kuan Han, Haiguang Wen, Junxing Shi, Kun-Han Lu, Yizhen Zhang, Di Fu, and Zhongming Liu. Variational autoencoder: An unsupervised model for encoding and decoding fmri activity in visual cortex. Neurolmage, 198:125-33, 2019.
James V Haxby, J Swaroop Guntupalli, Andrew C Connolly, Yaroslav O Halchenko, Bryan R Con- roy, M Ida Gobbini, Michael Hanke, and Peter J Ramadge. A common, high-dimensional model of the representational space in human ventral temporal cortex. Neuron. 72(2):404-416 2011.
Tomoyasu Horikawa and Yukiyasu Kamitani. Hierarchical neural representation of dreamed ob- jects revealed by brain decoding with deep neural network features. Frontiers in Computational Neuroscience, 11:4, 2017.
Tomoyasu Horikawa, Masako Tamaki, Yoichi Miyawaki, and Yukiyasu Kamitani. Neural decoding of visual imagery during sleep. Science, 340(6132):639 642, 2013.
Jingyang Huo, Yikai Wang, Yun Wang, Xuelin Qian, Chong Li, Yanwei Fu, and Jianfeng Feng. Neuropictor: Refining fmri-to-image reconstruction via multi-individual pretraining and multi- level modulation. In European Conference on Computer Vision, pp. 56-73. Springer, 2024.
Mason Kamb and Surya Ganguli. An analytic theory of creativity in convolutional diffusion models. In Proceedings of the International Conference on Machine Learning (ICML), 2025.
Eric R. Kandel, James H. Schwartz, Thomas M. Jessell, Steven A. Siegelbaum, and A.J. Hudspeth. Principles of Neural Science. McGraw-Hill Education, New York, 5th edition, 2013. ISBN 97800713901
Kendrick N Kay, Thomas Naselaris, Ryan Prenger, and Jack Gallant, Identifying natural images from human brain activity. Nature, 452(7185):352-33) 2008.
Sikun Lin, Thomas Sprague, and Ambuj K Singh. Mind reader: Reconstructing complex images from brain activities. Advances in Neural Information Processing Systems, 35:29624-29611 2022.
Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Yunfeng Lin, Jiangbei Li, and Hanjing Wang. Denn-gan: Reconstructing realistic image from fmri. In 2019 16th International Conference on Machine Vision Applications (MVA), pp. 1-6, IEEE, 2019.
Yulong Liu, Yongqiang Ma. Guibo Zhu, Haodong Jing, and Nanning Zheng. See through their minds: Learning transferable brain decoding models from cross-subject fmri. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pp. 5730-5738, 2025.
Tijana et al. Milekovic. Stable long-term bei-enabled communication in als and locked-in syndrome using lfp signals. Nature Communications, 911):1-13, 2018.
Martin M. Monti, Audrey Vanhaudenhuyse, Miriam R. Coleman, Melanie Boly, John D. Pickard, Léo Tshibanda, Adrian M. Owen, and Steven Laureys. Willful modulation of brain activity in disorders of consciousness. New England Journal of Medicine, 362(():579-589, 2010.
Milad Mozafari, Leila Reddy, and Rufin VanRullen. Reconstructing natural scenes from fmri pat- terns using bigbigan. In 2020 International joint conference on neural networks (IJCNN), pp- 1-8. IEEE, 2020.
Lorina Naci, Martin M. Monti, Damian Cruse, Andrea Kübler, Bettina Sorger, Rainer Goebel, Boris Kotchoubey, and Adrian M. Owen. Brain computer interfaces for communication with nonre- sponsive patients. Annals of Neurology, 72(3)311-323, 2012.
12

---

Published as a conference paper at ICLR 2026
Thomas Naselaris, Ryan Prenger, Kendrick N Kay, Michael Oliver, and Jack L Gallant. Bayesian reconstruction of natural images from human brain activity. Neuron, 63(6):902-99)) 915, 2009.
Shinji Nishimoto, An T Vu, Thomas Naselaris, Yuval Benjamini, Bin Yu, and Jack L Gallant. Re- constructing visual experiences from brain activity evoked by natural movies. Current biology, 21(19):164111646 2011.
Adrian M. Owen, Miriam R. Coleman, Melanie Boly, Matthew H. Davis, Steven Laureys, and John D. Pickard. Detecting awareness in the vegetative state. Science, 313(5592):1402,, 2006.
Furkan Ozcelik and Rufin VanRullen. Natural scene reconstruction from fmri signals using genera- tive latent diffusion. Scientific Reports, 13(1):15666, 2023.
Joel Pearson, Thomas Naselaris, Emily A. Holmes, and Stephen M. Kosslyn. Mental imagery: functional mechanisms and clinical applications. Trends in Cognitive Sciences, 19(10)590-602, 2015.
Kai Qiao, Jian Chen, Linyuan Wang, Chi Zhang, Li Tong, and Bin Yan. Biggan-based bayesian reconstruction of natural images from human brain activity. Neuroscience, 444:92-105, 2020.
Ziqi Ren, Jie Li, Xuetong Xue, Xin Li, Fan Yang. Zhicheng Jiao, and Xinbo Gao. Reconstruct- ing seen image from brain activity by visually-guided cognitive representation and adversarial learning. Neurolmage, 228: 17602, 2021.
Alexander Schaefer, Kong, Evan M Gordon, Timothy Laumann, Xi-Nian Zuo, Avram Holmes, Simon B Eickhoff, and BT Thomas Yeo. Local-global parcellation of the human cerebral cortex from intrinsic functional connectivity mri. Cerebral cortex, 2809):309533114, 2018.
Paul Scotti, Atmadeep Banerjee, Jimmie Goode, Stepan Shabalin, Alex Nguyen, Aidan Dempster, Nathalie Verlinde, Elad Yundler, David Weisberg, Kenneth Norman, et al. Reconstructing the mind's eye: fmri-to-image with contrastive learning and diffusion priors. Advances in Neural Information Processing Systems, 36:24705- 2023.
Paul S Scotti, Mihir Tripathy, Cesar Kadir Torrico Villanueva, Reese Kneeland, Tong Chen, Ashutosh Narang, Charan Santhirascgaran, Jonathan Xu, Thomas Naselaris, Kenneth A Norman, et al. Mindeye2: Shared-subjed models enable fmri-to-image with I hour of data. arXiv preprint arXiv:2403. 2007 2024.
Katja Sceliger, Umut Güçlü, Luca Ambrogioni, Yagmur Güşlütürk, and Marcel AJ Van Gerven. Generative adversarial networks for reconstructing natural images from brain activity. Neurolm- age, 181:775-785, 2018.
Guobin Shen, Dongeheng Zhao, Xiang He, Linghao Feng. Yiting Dong, Jihang Wang, Qian Zhang, and Yi Zeng. Neuro-vision language: Enhancing brain recording based visual reconstruction and language interaction. Advances in Neural Information Processing Systems, 37:98083 981000 2024.
Guohua Shen, Tomoyasu Horikawa, Kei Majima, and Yukiyasu Kamitani. Deep image reconstruc- tion from human brain activity. PLoS computational biology, 15(1):e 2019.
Yunsheng Shi, Zhengjie Huang. Shikun Feng, Hui Zhong, Wenjing Wang, and Yu Sun. Masked label prediction: Unified message passing model for semi-supervised classification. In Proceedings of the Thirtieth International Joint Conference on Artificial Intelligence (IJCAI-21), pp. 1548- 1554. International Joint Conferences on Artificial Intelligence Organization, August 2021. doi: 10.24963/jjcai.22211 2021/214. URL httltttts://ddol /ttt:::://ooll.orr/00..//////// (ttp://311.a....2211.......
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Ghislain St-Yves and Thomas Naselaris. Generative adversarial networks conditioned on brain activity reconstruct seen images. 2018 IEEE international conference on systems, man, and cybernetics (SMC), pp. 1054-1061. IEFE, 2018.
13

---

Published as a conference paper at ICLR 2026
Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14453-14463 2023.
Mingxing Tan. Efficientnet: Rethinking model scaling for convolutional neural networks. arXiv preprint arXiv:1905.1 /9946, pp. 6105-6114, 2019.
Dmitry Ulyanov, Andrea Vedaldi, and Victor Lempitsky. Deep image prior. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9446 9446-9454, 2018.
Aaron van den Oord, Yazhe Li, and Oriol Vinyals. Representation learning with contrastive pre- dictive coding. In Advances in Neural Information Processing Systems, 2018. URL https:
http:/bbb11007.007885
Shizun Wang. Songhua Liu, Zhenxiong Tan, and Xinchao Wang. Mindbridge: A cross-subject brain decoding framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1 11333-11342, 2024a.
Zhou Wang, A.C. Bovik, H.R. Sheikh, and E.P. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600-612, 2004. doi: 10.1 10.1109/T1P220008888999
Zicheng Wang. Zhen Zhao, Luping Zhou, and Parashkey Nachev. Unibrain: A unified model for cross-subject brain decoding. arXiv preprint arXiv:2412.19487, 2024b.
Navve Wasserman, Roman Beliy, Roy Urbach, and Michal Irani. Functional brain-to-brain transfor- mation with no shared data. arXiv preprint arXiv:2204.1 11100, 2024.
Weihao Xia, Raoul De Charette, Cengiz Oztireli, and Jing-Hao Xue. Dream: Visual decoding from reversing human visual system. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 8226-8235, 2024a.
Weihao Xia, Raoul de Charette, Cengiz Oztireli, and Jing-Hao Xue. Umbrae: Unified multimodal brain decoding. In European Conference Computer Vision, pp. 242 259. Springer, 2024b.
Kentaro Yamada, Yoichi Miyawaki, and Yukiyasu Kamitani. Inter-subject neural code converter for visual image representation. Neurolmage, 113:289-297, 2015.
Chi Zhang, Kai Qiao, Linyuan Wang, Li Tong, Ying Zeng, and Bin Yan. Constraint-free natural im- age reconstruction from fmri signals based on convolutional neural network. Frontiers human neuroscience, 12:242, 2018a.
Richard Zhang, Phillip Isola, Alexei A. Efros, Eli Shechtman, and Oliver Wang. The unreasonable effectiveness of deep features as a perceptual metric. In Proceedings of the IEEE/CVE Conference Computer Vision and Pattern Recognition (CVPR), pp. 586-555 IEEE Computer Society, 2018b. doi: 10.1119/CVPP.22 000068. URL ht ptttts://arxxxr //tpxva.ivv///////a//eaa/aawaava wttp://.rrggb////aaoaoooo/ss abs 1801 03924
14

---

Published as a conference paper at ICLR 2026
Appendix
The appendix is organized into four sections. Sec. A presents ablations and analysis, while Sec.
B
and Sec. C provide further results. Specifically, Sec. B reports additional evaluation metries for
our reconstructions, and Sec. C includes many more visual reconstructions, showcasing additional images and comparisons for both the full training and transfer learning pipelines. Finally, Sec.
D
describes further technical details.
A A ABLATIONS AND ANALYSIS
We deseribe additional experiments analyzing different components and factors of our method. We begin with an analysis of the contribution of External Images. followed by an evaluation of different Clustering Methods, and finally examine the effect of the Number of Clusters.
A.1 INFLUENCE OF UNLABELED IMAGES
To further assess the impact of enlarging the training dataset, we evaluate the role of external images in our framework. As described in Sec. 3] training is based on paired fMRI-image samples from the NSD dataset. To mitigate the limited subject-specific data, we additionally use 1220 natural images from the unlabeled portion of COCO dataset with predicted fMRI responses generated by the mmage-to-fMRe encoder of Belly et al. (2024). As shown in Table TT] incorporating external images provides improvements across most evaluation metries, further supporting our choice to enlarge the training data in this way.
Low-Level Metries
High-Level Metrics
Method
PixCorr 1 SSIM f Alex(2) f Alex(5) 1 Incep CLIP t Eff + SwAV +
W/o External Images
0.365
0.494
97.5%
99.1%
96.5%
95.2% 0.592 0.340
With External Images
0.386
0.486
98.4%
99.5%
97.3% 96.4% % 0.564 0.320
Table TI: Ablation study on the effect of using external images during training. Incorporating predicted IMRI responses improves performance almost all evaluation metrics, highlighting the benefit of augmenting the limited subject-specing training data with additional unlabeled images.
A.2 FUNCTIONAL VS. ANATOMICAL CLUSTERING
Our method relies functional clustering, assigning voxels to clusters based encoding embed- dings from the Universal Encoder. As an anatomical alternative, we aligned all subjects to FSaverage space and clustered voxels using either (i) spatial 3D coordinates (ii) Schaefer atlas- cortical parcellations (e.g., Schaefer 7 networks Schaefer et al. (2018)). For the subset of voxels used here, the Schaefer-100 1000 atlas yielded 198 clusters, and the Schaeler-400 atlas yielded 84 clusters. This demonstrates our framework's 's compatibility with multiple clustering strategies, though functional clustering provides superior reconstruction performance.
Low-Level
High-Level
Method
PixCorr t SSIM t Alex(2) 1 Alex(5) 1 Incep 1 CLIP 1 Eff 4 SWAV +
-D-oordiisates clustering
0.378
0.491
98.1%
99.3%
96.6%
95.6% 0.589 0.336
Schaefer-400 parcellation
0.371
0.477
98.3%
99.3%
97.0%
96.1% 0.576
0.333
Schaefer- 1000 parcellation
0.378
0.475
98.7%
99.4%
97.4%
96.3% 0.572
0.331
Functional clustering
0.386
0.486
98.4%
99.5%
97.3%
96.4% 0.564 0.320
Table T2: Quantitative comparison Functional Anatomical clustering, Results average across subjects 1, 2, 5, and from the Natural Scenes Dataset.
15

---

Published as a conference paper at ICLR 2026
A.3 EFFECT OF CLUSTERS NUMBER
The number of clusters is hyperparameter our method. Here we show how varying this number affects performance. The results indicate that our approach is largely robust to this choice: perfor- mance remains strong across a wide range, provided the cluster count is not too small.
Low-Level
High-Level
Clusters
PixCorr
SSIM +
Alex(2) 1 Alex(5) t
Incep 1
CLIP f
Eff 4
SWAV J
8
0.379
0.492
97.6%
99.1% 1%
96.6%
95.6%
0.588
0.334
16
0.380
0.491
97.8%
99.3%
96.6%
95.6%
0.582
0.334
32
0.383
0.486
98.2%
99.3%
97.0%
96.0%
0.576
0.325
64
0.385
0.486
98.3%
99.5%
97.2%
96.1%
0.569
0.321
128
0.386
0,486
98.4%
99.5%
97.3%
96.4%
0.564
0.320
256
0.391
0.487
98.6%
99.5%
97.6%
96.3%
0.565
0.321
512
0.392
0.491
98.5%
99.5%
97.4%
96.1%
0.569
0.322
Table T3: Quantitative comparison across cluster numbers. The model is relatively robust to the number voxel clusters, with a slight decrease in performance for small numbers cluster.
A.4 REFINEMENT STAGE VISUAL RESULTS
we apply a refinement stage in Scotti et al. (2024) in which the image is passed through pre- trained SDXL diffusion model. This refinement improves overall realism, smooths textures, and corrects facial details, producing visually enhanced reconstructions, The figure shows that the addi- tional refinement slightly improves perceptual quality the generated images.
Seen
Before Refinement Refinement
After
Seen Image
Before Refinement Refinement
After
Image
Figure S1:
Refinement stage using a pre-trained SDXL diffusion model. The SDXL model enhances global realism, smooths textures, and corrects fine details such as the cat eyes, resulting in more natural and visually coherent reconstructions.
16

---

Published as a conference paper at ICLR 2026
A.5 TWO-BRANCH CONTRIBUTION
We evaluate the semantic and low-level branches individually and together. The semantic branch captures image content. The low-level branch preserves coarse outlines and structural detail. Quan- titatively, the low-level branch performs better on structural fidelity metrics such as pixel correlation and SSIM, while the semantic branch leads perceptual measures. Combining them yields reconstruc- tions that are both semantically accurate and structurally faithful, confirming their complementary contributions.
Low-Level Metrics
High-LLvel Metries
Method
PixCorr t
SSIM f
Alex(2) f Alex(5) Incep f CLIP
Eff 4 SwAV +
Low level branch
0.438
0.505
97.4%
99.1%
95.9%
85.8%
0.688
0.480
Semantic branch
0.336
0.431
98.5%
99.4%
96.1%
95.2%
0.602
0.345
Combined
0.386
0.486
98.4%
99.5%
97.3%
96.4%
0.564
0.320
Table T4: Branch contributions. Quantitative results for the semantic branch, the low-level branch, and the combined full pipeline. The low-level branch excels in structural fidelity metrics, while the full pipeline achieves balanced performance across both semantic and structural metrics.
Low-Level
High-Level
Method
PixCorr f SSIM 1 Alex(2) 1 Alex(5) 1 Incep f CLIP 1 Eff 4. SWAV +
BIT
0.500
0.518
99.4%
99.8%
96.8 % 87.5% % 0.667 0.462
MLP- InfoNCE loss
0.267
0.481
80.9%
85.0%
77.5%
69.1%
0.897 0.613
MLP MSE loss
0.182
0.520
54.2%
54.2%
52.0%
51.1% 0.976 0.670
Table T5: Quantitative comparison between BIT and MLP in the low-level branch. We assess the contribution of the Brain-Interaction Transformer (BIT) low-level feature reconstruction by comparing it against multilayer perceptron (MLP IK hidden dim) baseline. The results show that BIT substantially outperforms the MLP predictor in reconstructing low-level visual features. Results are for Subject 1.
17

---

Published as a conference paper at ICLR 2026
B ADDITIONAL QUANTITATIVE RESULTS
We present additional in-depth quantitative results. First, we report quantitative results on less satu- rated metrics than the ones shown in (Tabblee) Second, we present quantitative results on transfer learning reconstructions with less than hour of subject-specific data (Fig. S3
Additional Metrics Many of the metrics commonly reported for fMRI image reconstruction ap- pear saturated, suggesting little room for improvement and only small differences between methods. it largely stems from the of 2-way retrieval met-
However, this impression is misleading, rics and does not imply that the reconstruction problem is close to being fully solved. Conversely, some of the unsaturated standard metrics also be misleading for other reasons. For example, the structural similarity index (Wang et al., (2004). when evaluated on grayscale images, fails to capture important chromatic information. To address this, evaluate additional metrics that reveal sub- stantial room for progress and highlight a significant performance gap between Brain-IT and other methods: (i) 1000-way retrieval using CLIP embeddings of reconstructed versus ground-truth im- ages; (ii) LPIPS (Zhang et al. 2018b) using AlexNet Features, reflecting perceptual similarity which correlates well with human judgment; (iii) Color-SSIM- evaluating SSIM on the RGB images.
Method
SSIM-color 1 1000way-CLIP 1 LPIPS +
NeuroPictor
0.375
0.255
0.628
MindEye2
0.383
0.238
0.624
MindTuner 0.371
0.231
0.638
BrainIT(ours) 0.472
0.393
0.596
Table T6: Additional quantitative metrics. The additional metrics are evaluated on the reconstruc-
tions of subject 1, showing that when using non-saturated metries, big performance gap is revealed between Brain-ITT and previous methods.
Transfer learning very limited data. In Table T7 present quantitative results of reconstruc- tions generated via transfer learning to subject with varying amounts of subject-specific data-1 hour, 30 minutes, and 15 minutes. Complementing the qualitative results in Fig. S3 the quantitative evaluations demonstrate that Brain-IT effectively leverages its pretrained knowledge to adapt to a new subject, achieving strong performance even with as little 15 minutes of subject-specifie data (450 IMRI samples). To the best of our knowledge, this is the first time reconstructions from fMRI using less than one hour of subject-specifie data are qualitatively and quantitatively comparable to those obtained with 40 hours of data in prior prominent methods.
Method
Low-Level
High-Level
PixCorr SSIM f Alex(2) Alex(5)
Incep
CLIP Ef' 4 SwAV +
BrainIT min)
0.336
0.476
97.6%
98.4%
90.7% 91.3% 0.706 0.404
BrainIT (30 min)
0.378
0.480
99.1%
99.3%
94.3%
93.3% 0.652 0.372
BrainIT (1 hour)
0.396
0.484
99.4%
99.6%
95.9%
94.7% 0.619 0.348
Quantitative evaluations of transfer learning to subject using different amounts of training data. We show that even using 15 and 30 minutes of subject-specific data, Brain-IT produces reconstructions on par with previous methods of transfer learning.
Table T7:
18

---

Published as a conference paper at ICLR 2026
C ADDITIONAL QUALITATIVE RESULTS
C.I RECONSTRUCTIONS WITH 40 HOURS OF DATA
Seen Image
Brain-IT
Mind
Neuro VLA
Neuro Pictor
(Ours)
Tuner
MindEye2
UMBRAE
Figure S2: Qualitative reconstruction results for subject 1. Comparison to prominent previous methods on large amount of images demonstrates the visual faithfulness of Brain-IT, alongside semantic accuracy.
19

---

Published as a conference paper at ICLR 2026
Seen Image
Brain-ITT (Ours)
Mind Tuner
Neuro VLA
Neuro Pictor
MindEye2
UMBRAE
Figure S2: Qualitative reconstruction results for subject 1- Part
20

---

Published as a conference paper at ICLR 2026
Seen Image
Brain-IT (Ours)
Mind Tuner
Neuro VIA
Neuro Pictor
EendEye2
UMBRAE
STOP
STOP
STOIP
STOP
STTP
STOP
CTTUI
Figure S2: Qualitative reconstruction results for subject 1- Part
21

---

Published as a conference paper at ICLR 2026
Seen Brain-IT Image
Mind Tuner MindEye2
Neuro VIA
Neuro Pictor
(Ours)
UMBRAE
Figure S2: Qualitative reconstruction results for subject 1- Part
22

---

Published as a conference paper at ICLR 2026
C.2 RECONSTRUCTIONS WITH TRANSFR--LEARNNG ON LIMITED DATA
Brain-IT (Ours)
Eyn
MindTuner
Seen
Image
15 min
30 min
hour 40 hours hour 40 hours
hour 40 hours
Figure S3: Qualitative reconstruction results for transfer learning to subject with limited data. Brain-IT reconstructions using 1 hours and even 30 minutes are comparable to reconstructions of previous methods generated using 40 hours of data.
23

---

Published as a conference paper at ICLR 2026
Brain-IT (Ours)
MindEyc2
MindTuner
Seen
Image
min 30 min
hour 40 hours hour 40 hours
hour 40 hours
Figure S3: Qualitative reconstruction results for transfer learning with limited data Part
24

---

Published as a conference paper at ICLR 2026
Brain-IT (Ours)
MyndEyc2
MindTuner
Seen
Image
min 30 min hour 40 hours hour 40 hours
hour 40 hours
Figure S3: Qualitative reconstruction results for transfer learning with limited data Part
25

---

Published as a conference paper at ICLR 2026
Brain-IT (Ours)
MyndEyc2
MindTuner
Seen
Image
min 30 min 1 hour 40 hours hour 40 hours
hour 40 hours
Figure S3: Qualitative reconstruction results for transfer learning with limited data Part
26

---

Published as a conference paper at ICLR 2026
C.3
IMAGE RECONSTRUCTIONS FOR ADDITIONAL SUBJECTS
Seen Image
Seen Image
Sub
Sub 2
Sub 5
Sub 7
Sub 1
Sub
Sub
Sub 7
Figure S4: Image reconstruction results for subjects 1, 2, 5 and Due to its use of shared functional clusters and network modules, Brain-IT successfully leverages cross-subject information to produce effective reconstructions for four evaluated subjects
27

---

Published as a conference paper at ICLR 2026
Seen Image
Seen
Sub
Sub 2
Sub 5
Sub 7
Image
Sub 1
Sub 2
Sub
Sub 7
STOP
STOP
STOP
STOP
STOP
Figure S4: Image reconstruction results for subjects 1, 5 and 7 Part B
28

---

Published as a conference paper at ICLR 2026
C.4 BRAIN TOKENS SPATIAL AND SEMANTIC SELECTIVITY
In Brain-IT, each IMRI voxel cluster is represented by learned brain token, and each predicted image feature is associated with a learned query token. The brain tokens encode the functional activity patterns of specific voxel clusters, while the query tokens index the image features that the model must predict. These two sets of tokens interact through the rross-attention layers of the Crain-Interaction Transformer (BIT): for each query token, BIT attends over all brain tokens and aggregates their information to predict the image feature corresponding to that query.
Each query token is tied to a specific spatial position in an image feature map, so that every predicted feature can be localized in the image. Concretely, for the low-level branch, query tokens correspond to spatial positions in a VGG feature layer (an A x grid), and for the semantic branch they cor- respond to spatial tokens in the CLIP image encoder (e.g.. 16 x 16 grid of patch tokens). Our goal is to show how different brain tokens contribute to the prediction of different image features via these cross-attention maps, thereby revealing the direct flow of information from functionally defined voxel clusters (brain tokens) to localized image features (query tokens) used for reconstruc- tion. We present two complementary experiments: (i) brain-tokens that contribute features at specific spatial locations in the image (see Fig. 155; and (ii) brain-tokens that contribute to features capturing specific semantic concepts the image (see Fig. S6)..
Spatial selectivity of brain tokens. In the first experiment, we visualize the cross-attention map between brain tokens and query tokens by averaging the attention weights across all transformer layers and all attention heads. For each brain token, this yields a 2D attention map over query tokens whose axes correspond to the spatial layout of the image features (an 56 x 56 grid for VGG 1 1.2 layer, and a 16 x 16 grid of semantic spatial tokens). Each small square in Fig. [S5p shows the attention map for one brain token; we arrange these squares highlight tokens that consistently attend to specific image locations. In SS,, we assign a unique color to each brain token shown in Fig. S55,, and color all voxels its corresponding cluster with the same color. This reveals which brain regions most strongly contribute to predicting features at particular image locations. As expected, we observe clear contralateral organization: brain tokens in the right hemisphere predominantly support features in the left part of the image, and vice versa.
Semantic selectivity of brain tokens. In the second experiment, demonstrate semantic selec- tivity rather than pure spatial position. For each brain token and each image, the corresponding IMRI response induces a different attention map from that token all query tokens. We again av- erage attention across layers and heads to obtain an attention map per brain token and per image, and then examine, across many images, which spatial regions (and thus which semantic content in those regions) given brain token most strongly contributes to. We select brain tokens whose at- tention consistently highlights image regions with a clear semantic meaning, as determined by what typically appears in the attended parts of the images. Fig. S66] illluutttrtte such semantically selective brain tokens, including tokens that preferentially contribute to predicting features for hands, legs, text/words, faces, and other object parts. For each example, gions and the corresponding voxel cluster on the cortical surface, revealing where in the brain the voxel clusters that support these semantic predictions are located.
show both the attended image
29

---

Published as a conference paper at ICLR 2026
Brain tokens spatial localization
a
Attention maps of brain tokens to query tokens Each of the 6x6 maps corresponds to single brain token
VGG layerl
Semantic
(b) Voxel clusters assigned to brain tokens
Cluster color by
L
R
Attn-map position
DO
Figure S5: Brain Token Spatial Attention Maps (a) Brain tokens' average attention maps (6 x grid, 36 tokens shown), averaged across 1K test images. Each map demonstrates the spatial region in the reconstructed image that the corresponding brain token influences.
(b) Brain token brain maps. A color map assigns color to each brain token according to the position of it's attention map in the 6 6 grid (a). The brain map shows the voxels corresponding each brain token, with their color determined by assigned color. This demonstrates the link between token's anatomical origin and its attended image location.
30

---

Published as a conference paper at ICLR 2026
Brain tokens Semantic selectivity in BIT
Attention maps of brain tokens to query tokens
Kites & planes
Hands
Body Silhouette
Legs
Words
Faces
(b) Voxel cluster assigned to brain tokens
L
R
Figure S6: Semantic Selectivity of Brain Tokens (a) Semantic attention maps. Each image group showcases the average attention map of single brain token within the Brain-IT semantic branch, overlaid on the original images. This evaluated on the test images. We can observe clear semantic selectivity for each brain token (e.g., words, body silhouettes, faces). (b) Brain token brain maps. The brain map shows the anatomical source voxels of the tokens presented in (a), color-coded by the corresponding frame color in (a).
31

---

Published as a conference paper at ICLR 2026
C.5 RECONSTRUCTIONS ON NSD-SYNTHETIC
Seen Image
Seen Image
Seen Image
Recon
Recon
Recon
Stvding
Figure S7: NSD-Synthetic reconstructions results for Subject 1 on the NSD- SSD-Synthetic dataset Gif- ford et al. (2024), an extension of NSD with out-of-distribution stimuli. Brain-IT' applied without any adaptation, demonstrating robust generalization to out-of-distribution stimuli.
32

---

Published as a conference paper at ICLR 2026
C.6
TWO-BRANCH CONTRIBUTION IMAGE RECONSTRUCTIONS
Seen Image
Low level branch
Semantic branch
Both Branches
Seen Image
Low level branch
Semantic branch
Both Branches
Figure S8: Reconstructions via the two generation branches of Brain-IT. The combination of both the low-level and the semantic branch ensures both semantically and visually accurate recon- structions. Importantly, the low-level branch often leads to accurate modifications of the semantic content, while the semantic branch often carries important visual information.
C.7 FAILURE CASES
Seen Image
Brain-I'T
Seen Image
Brain-I'''
Seen Image
Brain-I'T'
Figure S9: Failure cases. For certain fMRI samples, Brain-IT will predict either the semantic content or the visual layout of the image relatively accurately, but will fail to predict the other accurately, leading to incorrect reconstructions.
33

---

Published as a conference paper at ICLR 2026
D ADDITIONAL TECHNICAL DETAILS
We provide additional details and explanations on training (Sec. D.1 the low-level branch (Sec. D.2), inference time generation (Sec. D.3), and transfer learning (Sec. D.4
D.1 TRAINING DETAILS
Voxel sampling During training we sample voxels both to reduce memory usage and as form of regularization. For each example, we uniformly sample 15K voxels with replacement from the 440K voxels of each subject. Our Brain Tokenizer, based on GNNs, can naturally handle variable number voxels, with the voxel-to-cluster mapping (GNN edge index) sampled consistently with the selected voxels.
Enriching training with unlabeled images To augment training, we use 1120 unlabeled images from the COCO unlabeled split. For each image, we predict synthetic fMRI using a Universal Encoder trained on the same training data, generating predictions for all eight subjects. During training. we iterate over all labeled NSD data and all unlabeled images in each epoch, and for each unlabeled image, we randomly sample the predicted fMRI of one subject as input.
Training parameters We train the low-level branch (VGG prediction) using InfoNCE loss and the semantic branch stage 1 (CLIP alignment) using L2 loss. Both models trained for 60 epochs using mixed-pre pixxed-prcission training and the Adam optimizer with learning rate of 5e-4 and a 15-epoch warmup. We apply ROntaaeL with factor of 0.1 based validation performance. The batch size is for lowelevel and 128 for semantic (stage 1). We hold out 10% of the training set for validation tune hyperparameters and select the best checkpoint.
Second stage of semantic branch (joint training ii trained with mixed precision using a batch size of 16 and gradient accumulation over 4 steps. We use the AdamW optimizer with a learning rate of le-5, and train for 10 epochs on images of size 256x256.
Training times and resources Training times on H100 GPUs were: low-level branch 12 (1 GPU); semantic branch stage 1: 8 (1 GPU); stage 2: 10 (4 GPUs).
D.2 LOW-LEVEL BRANCH
Our low-level image reconstruction framework uses the DIP framework together with the VGG fea- tures predicted by the BIT model to produce low-level image corresponding to the fMRI brain activity. This low-level image is then integrated into the diffusion process, resulting in reconstruc- tions that are both semantically and structurally faithful to the actual seen image.
VGG features We use pretrained VGG-16 16 network with batch normalization to extract features. Features are computed from 112 112 images to reduce the number of positions to predict. We extract features from VGG layers 1_2, 2_2, 3_3, 4_3, and 5_3. For layers 1_2 and 2_2, we merge 2 x 2 adjacent positions into a single token by concatenation (1.2 without overlaps and 2.2 with overlaps), while for the remaining layers each position corresponds to token. This results in 562, 552, 28a, 142, and 72 tokens for layers through 5, respectively. The InfoNCE loss is applied to each layer separately. Different VGG layers have different numbers of channels, resulting in tokens of varying dimensionality. Tokens with fewer than 512 dimensions are replicated until reaching size 512. During training we randomly sample which positions (tokens) predict for each layer. Specifically, we sample 512, 512, 128, 64, and 16 tokens from layers through 5, respectively, in order to reduce memory usage. At inference time, we predict all tokens and reshape them back into convolutional layers, with overlapping positions (in layer 2) averaged.
DIP inversion We use a DIP model based on U-Net with input dimensionality 32, internal dimensionality 128, and 3 scales with bilinear interpolation. Training is performed for 2K iterations with initial noise of 0. regularization noise of 1 1/30, and an exponential moving average with factor 0.99 applied to the output. All VGG layers assigned equal weight during inversion. The inverted image has the same resolution as the VGG input (112 x 112), and is upsampled to 256 x 256 to initialize the diffusion model.
34

---

Published as a conference paper at ICLR 2026
D.3 INFERENCE TIME GENERATION
To integrate the low-level image information with the semantic branch, we adopted an image-to- image configuration of the UnCLIP-based Stable Diffusion model. Instead of starting from pure Gaussian noise, we initialized the diffusion process with a partially noised version of the low-level branch output (the DIP inversion result), allowing the model retain structural information while refining semantic content. The diffusion sampler performed 38 denoising steps in total, beginning from an intermediate noise level equivalent to step 14 in the noise schedule. Starting from this level adds a moderate amount of noise to the input latent, enough to enable semantic refinement, while still preserving much of the original structure.
D.4 TRANSFER LEARNING TRAINING DETAILS
The transfer learning process consists of two steps: (i) We first adapt the "Brain Encoder" to the new subject as described in Beliy et al. (0024) (which can also be done with little training data). The resulting Encoder voxel embeddings Voxels-to-Clusters (V2C) Mapping, and to initialize BIT's Decoder voxel embeddings. (ii) In the second step, we apply the Brain-IT training procedure to update the voxel embeddings of the new subject, while freezing all other components. Training is performed with the limited subject-specifie samples, supplemented by predicted fMRI responses (on unlabeled data) from the adapted encoder in Step (i). This step mirrors the sequence of the original training pipeline (see Sec. 3 which include the two part: (i) Feature Alignment and (ii) Joint Training.
used both to assign each voxel a cluster in the
D.5 EVALUATION AGAINST OTHER METHODS
We compare Brain-ITT against several prominent fMRI-to-image reconstruction methods. For quan- titative evaluation, we rely on previously published results reported by these methods. We follow the standard protocol adopted in prior NSD reconstruction papers: using the shared images seen by all subjects as the test set. Note that the full 40 sessions of NSD were released only after the com- pletion of the 2023 Algonauts Challenge. Consequently, some methods were trained on 37 instead of 40 sessions (a relatively minor difference) and evaluated on 982 test images rather than the full 1000. MindEye22 recalculated evaluation metrics for several prominent baselines using their avail- able code. All our experiments were trained the full 40 sessions and evaluated on the complete test set. In our reported quantitative results, the following methods correspond to evaluations on 982 test images (rather than 1000): DREAM, UMBRAE, NeuroVLA. MindBridge and NeuroPictor. For qualitative comparisons and for computing additional metries, we used reconstructed images provided by top-ped pop-performing methods (some publicly available, others shared upon request).
35


Reviews:
[{"review_id": "SpvuNxpBrj", "reviewer": "Reviewer_zwfk", "summary": "The paper introduces Brain-IT, a new framework for reconstructing natural images from fMRI data. Its key innovation is the Brain Interaction Transformer (BIT), which introduces functionally shared voxel clusters across subjects, mapped via a Gaussian Mixture Model over voxel embeddings derived from a pre-trained “Universal Brain Encoder.” Results show strong performance on both structural (PixCorr, SSIM) and perceptual (CLIP, Inception, AlexNet) metrics outperforming prior SoTA methods. Brain-IT sets a very strong benchmark for fMRI-Image decoding pipelines with SoTA performance in full and limited data settings. Brain-IT achieves comparable results to full-data baselines with only 1 hour of subject-specific data, and qualitatively plausible reconstructions even with 15 minutes.", "questions": "1. Verification of Low-Level Pipeline and VGG Model Details: The low-level reconstruction results are exceptionally strong with structural metrics (SSIM, PixCorr) being high while simultaneously maintaining strong semantic metrics (CLIP+Inception>85%), which is unusual since prior low-level pipelines typically show a trade-off (almost all previous works' LL pipelines have CLIP,Inception around the 50-60s). Given the novelty and importance of this component can you clarify the following:\n- Which specific pretrained VGG checkpoint is used?\n- Can you confirm there is no train/test overlap between the VGG pretraining data and COCO images used in NSD?\n\n2. **Ablation on Low-Level Pipeline Contribution:** Another question regarding the low level pipeline. I believe your strong gains in the LL pipeline may be the primary driver of performance gains. Can you provide additional ablation experiments to isolate teh contribution of V2C+BIT from the VGG+DIP pipeline? For instance:\n- Train an ME2 model on a single subject but switch the low level ground truth to VGG+DIP. Use an MLP similar to previous works feeding in the entire fMRI instead of doing V2C.\nAnything along these lines to clarify whether the gains primarily come from the novel architectural choices (V2C mapping, BIT interactions) or the low-level reconstruction approach would be helpful.\n\n3. **Loss Function Design Choices:**  You use the L2 loss for Feature Alignment and standard diffusion loss for Joint Training. This differs from recent works that employ contrastive losses, multiple alignment objectives, or diffusion priors to align fMRI with CLIP embeddings.\n- Have you experimented with contrastive losses (e.g., InfoNCE) or other alignment objectives for the CLIP prediction task?\n- What motivated using only L2 loss for the initial alignment stage?\n- Could you report intermediate alignment metrics (e.g., cosine similarity between predicted and ground-truth CLIP embeddings) after each semantic training stage to better understand the alignment during the two stages?\n\n4. Do you use image-to-text captioning (as in ME2) during the SDXL refinement stage, or only the CLIP-based conditioning?\n\nAdditional:\nTypo in Figure 4: Trasnformer instead of Transformer\n\n\nOverall I believe this a strong contribution to the field and I am open to increasing my score if my concerns are addressed.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. **SoTA empirical results:** Outperforms all previous methods in full-data and limited-data settings.\n2. **Novel techniques:** The idea of functional voxel clustering for fMRI-Image decoding is novel and improves performance, finding an elegant way to incorporate transformers into the mapper architecture. \n3. **Extensive quantitative and qualitative evaluation:** The paper provides clear tables, broad baselines (MindEye, BrainDiffuser, MindTuner, MindEye2, NeuroVLA, etc.), and an ablation appendix exploring cluster number and anatomical vs functional clustering.\n4. **Use of additional synthetic training data:** While the idea of using synthetic signals to improve alignment has been considered in previous works, using it to generate significantly more training data is a neat idea that seems to provide positive gains.", "weaknesses": "1. **Ambiguity in Functional Clustering Implementation:** While the voxel-to-cluster mapping is conceptually clear, I would appreciate a clarification on a few details:\n- Are the voxel embeddings unique per-voxel-per-subject (i.e., one embedding vector per voxel that is fixed regardless of the input image), or do they vary per-image?\n- Are these voxel embeddings frozen after the initial V2C mapping is established, or are they continuously optimized during BIT training? Section 5.3 mentions that adapting to a new subject requires optimizing only the voxel embeddings, suggesting they are trainable, but the timeline of when they are optimized versus frozen could be more explicit.\n2. **Generalization to Out-of-Distribution Data:** The joint training approach in Stage (ii) of the semantic branch (Section 3.2) fine-tunes the diffusion model end-to-end with BIT on COCO-derived data. While you note this allows the models to \"establish a representation that is better suited for conditioning,\" this differs from prior works that keep the diffusion model frozen. This raises a question about generalization: Could you comment on how this joint training might affect the model's ability to generalize to fMRI data from substantially different experimental paradigms (e.g., THINGS dataset with different stimuli distributions, or datasets with different scanning protocols)? While the strong NSD results are impressive, while pushing the SoTA on 7T fMRI results is impressive, the more important problem in the field is building models capable of transferring to new scanning protocols or modalities (M/EEG). Developing a method that gives gains on an already good baseline at the cost of increased overhead for cross-dataset generalization might undermine the contribution. \n3. **Insufficient Evidence for \"Localized Image Features\" Claim:** Throughout the paper, you emphasize that BIT enables \"direct flow of information from brain-voxel clusters to localized image features\" as a key distinction from prior work. However, the predicted outputs are global CLIP embeddings (256 spatial tokens from ViT-bigG/14, which still represent the entire image) and VGG features. Could you provide evidence or analysis demonstrating that the functional clusters map to spatially localized regions in the predicted features? For instance, do specific clusters consistently predict features corresponding to particular spatial regions of images? Without such evidence, the \"localized\" claim appears to refer more to the intermediate representation (Brain Tokens) rather than the actual output features, which would make it similar to prior works in this regard.\n4. **Lack of codebase:** I believe this is the biggest weakness of this work. I understand that the authors promised to release the code and checkpoints upon acceptance. But when you claim to have SoTA empirical results in a rapidly growing field, it is vital to provide the code necessary to validate your results. The gains are significant from previous works and without the ability to verify these gains, it does leave room for some skepticism.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "LdgjMFbfE3", "reviewer": "Reviewer_P5xc", "summary": "The paper proposes Brain‑IT, a pipeline that maps fMRI signals to localized image features via a Brain‑Interaction Transformer (BIT) and reconstructs images through two complementary branches: a low‑level branch and a high‑level semantic branch. This design leverages coarse layout from the low‑level path to initialize diffusion and uses semantics to refine details. On NSD, the method achieves SOTA performance across most metrics, and demonstrates efficient transfer learning with only 15 minutes for new subject. The paper includes several ablations and states an intent to release code, checkpoints, and reconstructions.", "questions": "- During pretraining on NSD, did the authors exclude the shared 1,000 images, i.e., are the shared images strictly held out throughout? I would like to confirm no leakage (or not).\n- Related to the above question, I also would like to confirm transfer‑learning protocol and data usage.\n    - For a new subject, which stages use that novel subject’s data, and how much? Please clarify if any of the new subject’s data is used in the shared pretraining stages or any other training stages.\n    - For non‑target subjects (= training subjects), did the authors use all their data in all pretraining stages? Does this include or exclude the shared 1,000 images? \n- For anatomical clustering, the paper states that voxels are clustered by 3D coordinates in FSaverage space and then run through the same pipeline. However, I wasn’t fully sure how the anatomical clustering baseline was implemented. Could the authors provide more detail on the procedure?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- Brain‑IT outperforms prior methods on most of the conventional metrics in the 40‑hour setting and 1‑hour setting as well. It also reports first results for 15/30‑minute reconstruction on single subject. \n- Ablations cover usage of external unlabeled images, functional vs. anatomical clustering, and number of clusters, plus branch‑wise contributions. \n- They pledge to release code, checkpoints, and all reconstructed images upon publication.", "weaknesses": "- There is no explicit OOD evaluation (e.g., NSD‑synthetic) to probe robustness beyond the training distribution. \n- The paper primarily focuses on reconstruction accuracy and does not analyze what the Brain Tokens capture, thus provides limited neuroscientific insights.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "okZ6Y9DPv3", "reviewer": "Reviewer_todj", "summary": "This paper presents Brain-IT, a novel framework for reconstructing images from fMRI using a Brain Interaction Transformer (BIT). The method clusters functionally similar voxels across subjects to form shared functional units, which BIT maps to both semantic and structural image features for guiding a diffusion-based reconstruction. Experiments on the NSD show that Brain-IT achieves superior performance over existing methods in both qualitative and quantitative evaluations, while maintaining strong performance even with limited fMRI recordings.", "questions": "1. What specific scientific question does Brain-IT aim to address regarding brain function or visual processing?\n2. How does BIT capture or validate specific neural interaction patterns between brain regions?\n3. How does your V2C mapping differ technically and conceptually from Beliy et al. (2024), especially in achieving shared clusters?\n4. Why was DIP+VGG chosen over VAE-based reconstruction, and have you tested this choice empirically?\n5. Have you compared your functional clustering with established anatomical-functional parcellations such as Schaefer et al. (2018)? If not, why choose a coordinate-based baseline, and how might results change with a more realistic anatomical framework?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The brain-inspired BIT design is both biologically plausible and effective, leveraging shared functional voxel clusters to address data scarcity and inter-subject variability.\n* The dual-branch reconstruction (structural and semantic) balances visual fidelity and semantic accuracy, producing reconstructions that better match the original stimuli.\n* The efficient transfer learning strategy enables fast adaptation to new subjects with minimal fMRI data, greatly improving practicality.", "weaknesses": "* The paper lacks a clear distinction between its scientific and technical goals. While it emphasizes \"brain interaction\", it does not analyze what kinds of neural interactions BIT is meant to model or how these relate to known brain mechanisms, making the neuroscience contribution unclear.\n* The Voxel-to-Cluster (V2C) mapping largely follows Beliy et al. (2024) but does not clearly explain what is new. How the shared clusters differ from prior work and how they are optimized for decoding rather than encoding?\n* The low-level reconstruction choice (DIP+VGG) is insufficiently justified, with no comparison to standard alternatives like VAEs that could test whether the chosen method is actually superior.\n* The anatomical clustering baseline is overly simplistic, relying only on 3D coordinates rather than established anatomical-functional parcellations (e.g., Schaefer et al., 2018). This makes the comparison less meaningful, as the performance gap may reflect a weak baseline rather than a true advantage of functional clustering.\n\n[1] Schaefer et al. (2018). Local-Global Parcellation of the Human Cerebral Cortex from Intrinsic Functional Connectivity MRI.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)