Moving Beyond Diffusion: Hierarchy-to-Hierarchy Autoregression for fMRI-to-Image Reconstruction
Authors: Xu Zhang
Ruijie Quan
Wenguan Wang
Yi Yang
Venue: ICLR 2026 Poster (2026)
Keywords: fMRI-to-Image Reconstruction
Coarse-to-Fine Generation
Scale-wise Autoregressive Modeling
Scale-aware Neural Guidance
Primary area: applications to neuroscience & cognitive science
TL;DR: MindHier, a coarse-to-fine autoregressive framework, uses scale-aware guidance to inject hierarchical neural features for fMRI-to-image reconstruction, surpassing diffusion models in speed, stability, and semantic accuracy.

Abstract: Reconstructing visual stimuli from fMRI signals is a central challenge bridging machine learning and neuroscience. Recent diffusion-based methods typically map fMRI activity to a single neural embedding, using it as static guidance throughout the entire generation process. However, this fixed guidance collapses hierarchical neural information and is misaligned with the stage-dependent demands of image reconstruction. In response, we propose MindHier, a coarse-to-fine fMRI-to-image reconstruction framework built on scale-wise autoregressive modeling. MindHier introduces three components: a Hierarchical fMRI Encoder to extract multi-level neural embeddings, a Hierarchy-to-Hierarchy Alignment scheme to enforce layer-wise correspondence with CLIP features, and a Scale-Aware Coarse-to-Fine Neural Guidance strategy to inject these embeddings into autoregression at matching scales. These designs make MindHier an efficient and cognitively aligned alternative to diffusion-based methods by enabling a hierarchical reconstruction process that synthesizes global semantics before refining local details, akin to human visual perception. Extensive experiments on the NSD dataset show that MindHier achieves superior semantic fidelity, 4.67$\times$ faster inference, and more deterministic results than the diffusion-based baselines.

Published as a conference paper at ICLR 2026
MOVING BEYOND DIFFUSION: HIERARCHY- TO---RRRRCHYY AUTOREGRESSION FOR FMRI-TO-IME IMAGE RECONSTRUCTION
Xu Zhang 1.2, Ruijic Quan 1.2, Wenguan Wang Yi Yang!.2f
'The State Key Lab of Brain-Maching Intelligence, Zhejiang University, China ReLER CCAI, Zhejiang University, China
https: /tggt/ttthbbbgggg com/ http///hhanng///iiiiee.e
ABSTRACT
Reconstructing visual stimuli from fMRI signals is a central challenge bridg- ing machine learning and neuroscience. Recent diffusion-based methods typi- cally map fMRI activity single neural embedding, using it as static guidance throughout the entire generation process. However, this fixed guidance collapses hierarchical neural information and is misaligned with the stage-dependent de- mands of image reconstruction. In response, we propose MindHier, a coarse-to- fine IMRI-to-image reconstruction framework built on scale-wise autoregressive modeling. MindHier introduces three components: Hierarchical fMRI Encoder to extract multi-level neural embeddings, a Hierarchy-to- scheme to enforce layer-wise correspondence with CLIP features, and a Scale- Aware Coarse-to- Prarse-to-Finne Fine Neural Guidance strategy to inject these embeddings into autoregression matching scales. These designs make MindHier efficient and cognitively aligned alternative to diffusion-based methods by enabling a hierar- chical reconstruction process that synthesizes global semantics before refining lo- cal details, akin to human visual perception. Extensive experiments on the NSD dataset show that MindHier achieves superior semantic fidelity, 4.67x faster in- ference, and more deterministic results than the diffusion- based baselines.
Alignment
INTRODUCTION
Reconstructing visual stimuli from fMRI signals stands as fundamental challenge the intersec- tion of computer vision and cognitive neuroscience. Recent breakthroughs in this field have been largely driven by diffusion- et al., 2023; Rombach et al., 2022; Lu et al., 2022; Shen et al., 2026a; Fang et al., 2023; Quan al., 2024: Chen et al., 2025a; Sun et al., 2023; Huo et al., 2024; Chen et al., 2023b; Li et al., 2024; Song al., 2026; Shen al., 2026b), which typically align the encoded fMRI embedding with a single neural representation from multimodal models such as CLIP (Radford et al., 2021). Once aligned, this single neural feature serves as a fixed guidance signal throughout the entire diffusion process, progressively transforming random Gaussian noise into a reconstructed image (Fig. I(a)).
methods (Esser et al., 2021; Ho et al., 2020; Xue et al., 2024; Xu
Despite impressive results, relying solely on single, static neural feature guide the entire genera- tion pipeline introduces two fundamental limitations. First, fMRI signals are inherently hierarchical, in which different brain regions capture coarse semantic content as well as fine-grained perceptual details (Naselaris et al., 2009). However, current methods collapse this rich, multi-level information into a single vector, leading to its underutilization. Second, the guidance is temporally invariant, yet generative models operate in a dynamic multi-stage manner: early stages require global semantic constraints, while later stages demand precise structural and textural cues (Rissanen et al., 2023; Dieleman, 2024). Fixed neural guidance is often redundant in the early phase and insufficient in the later phase, creating a mismatch between representation and generation. Beyond these limitations, diffusion models themselves provide limited control points for injecting stage-awar guidance (Zhou et al., 2024), which hinders the effective exploitation hierarchical brain features.
Corresponding Author: Yi Yang.

---

Published as a conference paper at ICLR 2026
Nojse
(a)
Clear Image
(b)
Coarse
Diffusion Model
Seale- Wise--Wise Autoregressive Model
Neural Guidance
Scale-Aware Neural Guidance
Figure 1: Comparison of IMRI-to-image reconstruction pipelines. (a) Prior diffusion based methods utilize fixed neural feature to guide the reconstruction. (b) In contrast, MindHier employs scale- aware guidance, leveraging hierarchical neural features to first establish a low-resolution overview ("Forest") before progressively refining local details ("Trees") higher resolutions.
To address these challenges, propose MindHier, new coarse-to-fine IMRI-to-image reconstruc- tion framework built upon scale-wise autoregressive modeling (Tian et al., 2024), moving beyond diffusion-basee pipelines (Fig. 1(b)). MindHier integrates three key designs. (i) Hierarchical fMRI Encoder: a unified encoder that transforms fMRI signals into a hierarchy of embeddings, capturing the full spectrum of neural information from global semantics to local details. (ii) Hierarchy-to- Hierarchy Alignment: dual-objective training scheme that enforces sherarchytto-heraaae ment between fMRI embeddings and CLIP features, pairing shallow layers with low-level visual features for structural fidelity and deeper layers with high-level features for semantic coherence. (iii) Scale-Aware Coarse-to- Neural Guidance: a principled strategy that injects the hierarchi- cal embeddings into autoregression at matching scales. Specifically, the high-level semantic em- beddings from the fMRI encoder supervise the small-scale (low-resolution) stage to establish a se- mantically coherent global layout, while lower-level embeddings are progressively injected at larger scales (higher resolution) to refine structures and enrich textures. Together, these components enable MindHier to leverage the coarse-to- tonfine nature of visual autoregression, yielding reconstructions that achieve strong semantic fidelity and competitive structural faithfulness compared to diffusion-bart baselines. MindHier achieves state-of-the-ar high-level metrics the NSD dataset (Allen et al., 2022), e.g., the highest CLIP score 96.4% and the lowest SWAV distance of 0.329.
align-
Beyond empirical improvements, MindHier offers several deeper advantages. First, its coarse-to- fine reconstruction pipeline naturally echoes the "Forest before Trees" principle (Navon, 1977) from cognitive neuroscience, whereby human perception prioritizes global structure before resolving local details. By seeding reconstructions with high-level semantic embeddings and progressively refining fine-grained cues, MindHier translates this perceptual hierarchy into computational framework. Second, the coarse-to-fie autoregressive modeling substantially enhances efficiency: MindHier achieves significant speedups by allocating most computation to low-resolution scales. For exam- ple, reconstructing high-fidelity visual stimulus takes just 2.64 seconds per image, which is 4.67 faster than MindEye2 (Scotti et al., 2024). Third, MindHier produces more stable and consistent reconstructions. Unlike irftusion-based pipelines that are inherently stochastic due to random noise initialization, MindHier anchors its generation process directly to fMRI-derived embeddings, ini- tializing with high-level embeddings and conditioning subsequent scales on lower-level ones.
In nutshell, our contributions are three-fold:
We identify a fundamental mismatch between the fixed neural guidance and the dynamic nature image reconstruction. To address this, we introduce MindHier, a coarse-to-fin autoregressive framework that dynamically tailors neural guidance the specific demands of each generation stage, from establishing global semantic layout to rendering fine-grained details.
We propose Maaerchy-to-HHierrree Alignment disentangle the fMRI signal into hierarchical neural features, and Scale-Aware Guidance to strategically inject these features into the corre- sponding scales of the autoregressive generator for high-fidelity image reconstruction, resolving the structural mismatch between the neural representation and image reconstruction.
MindHier establishes critical balance among high semantic fidelity, deterministic stability, and inference speed. These advantages advance fMRI-to-image reconstruction beyond offline analy- sis, facilitating the future realization of real-time brain-computer interfaces.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
IMRI-to-Image Image Reconstruction. The reconstruction of visual stimuli from brain activity is piv- otal challenge at the intersection of cognitive neuroscience and computer vision. Early works (Kay et al., 2008; Miyawaki et al., 2008; Nishimoto et al., 2011) correlate fMRI signals with hand- crafted image features, while subsequent advances (Horikawa & Kamitani, 2017) improve this brain-visual association by using sparse linear regression predict features from early CNN lay- ers. The advent of Variational Autoencoders (Kingma & Welling, 2013) and Generative Adversarial Networks (Goodfellow et al., 2014) enables more direct pixel-level reconstruction (Seeliger et al., 2018; Ren et al., 2021; Ozcelik et al., 2022). However, these methods often yield ambiguous im- ages that lack clear semantics. To bridge this semantic gap, researchers have made notable progress by mapping brain signals to the latent space of models like IC-GAN (Perarnau et al., 2016) and StyleGAN (Karras et al., 2019). More recently, the field has been propelled by Latent Diffusion Models (Esser et al., 2021) and multi-modal models like CLIP (Radford et al., 2021). These state-of- the-art methods (Xia et al., 2024a; Ozcelik & VanRullen, 2023; Zeng et al., 2024; Gong et al., 2025; Scotti et al., 2023; 2024; Quan et al., 2024; Li et al., 2025b; Wang et al., 2024) map fMRI signals to the shared embedding space of CLIP to guide the reconstruction. However, these methods suffer from mismatch between the fixed guidance and the dynamic generation process. In light of this, we propose a distinct framework that adopts a coarse-to- generation process with scale-aware guidance, inherently preserving semantic consistency and structural faithfulness visual stimuli.
Visual Autoregressive Modeling. Autoregressive (AR) models have become a cornerstone of gen- erative modeling. Early visual AR approaches (Chen et al., 2020; Ramesh et al., 2021; Bai al., 2024) adapt the "next-token prediction" paradigm from language models. To do so, they employ discrete tokenizers (Van Den Oord et al., 2017) quantize images into ID sequence using a raster-scan (left to right, top to bottom). However, this process imposes an unnatural ID bias onto 2D images, often leading to suboptimal performance. A significant shift arrived with Visual Au- toregressive Modeling (VAR) (Tian et al., 2024), which introduces a novel "next-scale prediction" paradigm. This paradigm transforms an image into a pyramid of 2D token maps at different scales (resolutions) using a multi-scale residual VQ-VAE tokenizer. Each token map is generated progres- sively, conditioned on the previously generated ones. Such a paradigm better preserves spatial struc- ture and dramatically improves efficiency without sacrificing quality. Spurred by this innovation, recent works have applied this technique to diverse fields, including depth estimation (Wang et al., 2025), 3D generation (Zhang et al., 2024; Chen et al., 2025b), and text-to-image generation (Ma et al., 2024; Voronov et al., 2024; Han et al., 2025). Despite its proven power, the value of this paradigm for IMRI-to-image reconstruction remains underexplored. Building upon this paradigm, we propose a coarse-to-fine IMRI-to-image reconstruction framework with scale-aware guidance. Our framework reconstructs high-fidelity images and achieves inference speeds up to 4.67 x faster.
3 METHOD
We introduce MindHier for fMRIIto-image reconstruction, a framework built upon to-sse-t-ffine process, which can be broken down into three main components. First, we design a Hierarchical fMRI Encoder ($3.1) transform brain signals into a hierarchical set of neural features. Second, Alignment ($3.2) scheme to train this encoder, ensuring the features capture both finegrained details and global semantics. Finally, we detail the Scale- Aware Coarse-to- Fine Neural Guidance ($3.3), where these learned neural features are used to hierarchically guide a scale-wise autoregressive model during the reconstruction of visual stimuli.
we employ a
3.1 HIERARCHICAL FMRI ENCODER
The central challenge in fMRI-to-image reconstruction lies in bridging the representational mis- match between a static brain signal and the dynamic, multi-stage nature of image reconstruction. A naive approach relying on single feature would fail to capture this critical perceptual hierarchy. Such a feature provides redundant information for initial coarse synthesis while offering insufficient information for subsequent fine ien-iied refinement. Our solution is departure from this approach. We introduce a Hierarchical fMRI Encoder (HFE), an architecture designed not merely to encode the fMRI signal, but to disentangle it into structured multi-level representation. As illustrated in
3

---

Published as a conference paper at ICLR 2026
(a)
(b)
big reee stand in the sun and mist forest.
Cross Entropy Loss
L
Block
Mask
Ee. (1)
dx
Proj
Text Encnder
Layer fe01
Block M-1
Feod Forward
:1
Eq. (1)
Proj
Multi-Head Cruss, Attention
Crnca
Layer JM
Block
Multi- Self Attention
Proj
Eq. (5 (2)
AvgPool
Eq. (1)+
Start Token
Scale l-22
Scale k-1
Figure 2: Overview of the two-stage training pipeline of MindHier. (a) Stage 1: Hierarchy-to Hierarchy Alignment. A hierarchical fMRI encoder (composed of M cascaded blocks) is trained map fMRI signals feature hierarchy in CLIP space. This mapping is learned by aligning the encoder's outputs with corresponding intermediate features from a frozen CLIP vision encoder using a cascaded MSE loss (CMSE (Eq. 1)). To ensure high-level semantic coherence, the terminal IMRI feature is further aligned within CLIP's shared embedding space via a SoftCLIP loss (LsofCLip Neural Guidance. scale-wise autoregressive model is finetuned to generate images across K scales (coarse-to-fine fine), conditioned on features from the fixed fMRI encoder pretrained in Stage (a). practice, an attention mask is used to selectively route features via cross attention, which directs the features for the coarse view to attend to the initial scale and the features for finer details to guide subsequent scales. For illustration, simplified case with block count M 2, fMRI feature dimension CIMRI = 2, and scale count K is shown.
(Eq. 2)). (b) Stage Scale-Aware Coarse-to-
Fig. 2(a), the HFE is realized as a unified stack of M cascaded transformer blocks. Crucially, the outputs from these blocks, chical representation itself. This design leverages an established principle from Vision Transformers (ViTs) (Raghu et al., 2021) that ViT's progressively shift from processing local information in initial layers to aggregating purely global information in deeper layers. By explicitly aligning each block's output with a distinct level of pretrained vision model's feature hierarchy ($3.2), we compel the HFE to learn a structured decomposition of the neural signal. This process culminates in a cascade of features where the terminal output (eM) encapsulates the global, semantic "Forest," " while pre- ceding outputs (e,....) ) preserve the fine- fine-graine mee-grained "Trees". This tiered representation provides the dynamic, scale-aware guidance ($3.3) that is foundational to reconstruction framework.
are not only intermediate steps but constitute the hierar-
3.2 HIERARCCHYTT--HERRRRE ALIGNMENT
A hierarchical encoder necessitates a hierarchical training objective. To enable the HFE to disen- tangle the fMRI signal, we introduce composite loss function that enforces level-by-level corre- spondence between the emerging fMRI representations and the feature hierarchy of frozen CLIP vision encoder. This is achieved through two complementary objectives that are optimized jointly: one ensuring structural alignment and the other enforcing semantic coherence.
Structural Alignment. capture a rich hierarchy visual percepts, from simple textures to com- plex object parts, we enforce a direct, structural alignment between the intermediate representations of the HFE and the CLIP vision encoder. This implemented via a cascaded Mean Squared Error (MSE) loss, which promotes point-to-point correspondence between the feature sets at multiple levels of abstraction. For each of the M transformer blocks in the HFE, we minimize the distance between its output, Cm, and the feature map from corresponding vision encoder layer, Vgm.:
[l2((cm) ) (2 11.. M
LMSE
where 9m. is a mapping function from an fMRI block to a specific CLIP layer, and both fMRI features and visual features are l normalized for improving training stability (Yu et al., 2022).
4

---

Published as a conference paper at ICLR 2026
Semantic-Level Alignment. While structural alignment provides the necessary perceptual founda- tion, it lacks global semantic anchor to ensure the overall reconstruction is contextually correct. Since "Trees" are conceptually part of the "Forest", any initial error will be propagated through the whole reconstruction process. To address this, we supplement structural alignment with a contrastive objective bolster semantic alignment in CLIP's shared embedding space. Specifically, we align the terminal HFE feature, eM. representing the most abstract semantics, with both its visual feature Vau and its text caption feature t. This is accomplished using a SoftCLIP loss, which provides robust semantic anchor through both fMRI-to-image and fMRI-to-text alignment:
1 CSONCLIP = =-B
Σ 1=
log
exp(e: Vs/T) + log exple; t₃/T)) (imeeet//))))
exple, t./T)
(2)
Σ/= exp(e; V//T)
Ex=reppe;
where B is the batch size, T is a temperature hyperparameter. The explicit feature layer subscripts omitted for brevity. The total training objective for the fMRI encoder is the sum of the above loss terms, which trains the encoder to learn a hierarchical fMRI feature for subsequent reconstruction.
By enforcing hierarchical correspondence. this dual alignment ensures that information flows from CLIP to the hierarchical IMRI encoder in principled manner. The deep-to-deep alignment acts as conceptual "blueprint", establishing the overall layout first. The saallow-to-shalll alignment acts as fine-gre -naiigr "renderer", populating this blueprint with the precise textures, shapes, and struc- tural details perceived by the subject. This structured, top-down alignment is key reconstruction.
3.3 SCALE-AWARE COARSE-TO-FING NEURAL GUIDANCE
Given the hierarchical neural embeddings E= fc........ eM i our objective is to devise generation process that can leverage these structured embeddings. An ideal generative framework should there- fore be able accept distinet guidance signals at different stages of synthesis. To this end, we build upon scale-wise autoregressive modeling ("ian et al., 2024), a method uniquely suited for our task, as its "next-scale prediction" paradigm provides natural and discrete control points for our scale-aware guidance. In this paradigm, the fundamental regression unit is 2D token map at specific scale, where an input image is first mapped by an encoder & to a continuous representation f f= (I). and then discretized by quantizer Q into K multi-scale token maps R = Q(f) (ri,-... -' ). Each token map consists of discrete indices, where the index at a given position (i,j) determined by mapping its corresponding input vector to the closest entry in learned codebook Z E Rxx
(6.5 arg min (((()))) - lookup( E, n) 12 W (I.......N
Here, ((()) represents the input feature for the k-th quantization stage (where f1 = and subsequent inputs are residuals), N 4096 is the codebook size, and lookup(Z, nz) retrieves the n-th vector from the codebook. The standard autoregressive likelihood is then factorized across these scales, where each token map is generated conditional on all previously generated, lower-resolution maps:
121,,22,.00 FK)
II
K k- p(rk (1,,,,,..., Tk-1)
(4)
where each token map ra E
xwe is conditioned on the previously generated maps.
Our key innovation lies in how we condition this scale-wise generation process on our learned fMRI embeddings E= (ei...... }. We propose a novel, Scale-Aware Coarse-to-Fine Neural Guidance strategy where the generation of each token map Fı at scale & is conditioned not on a single, fixed embedding, but on a scale-specific conditional feature 8y from our fMRI hierarchy E:
p |
k= p(rk|T<k: 8k),
(5)
where Tick denotes the previously generated token maps. The guidance sy is dynamically selected from E according two cognitively inspired phases:
Seeding "Forest" (I: = 1): The initial, low-resolution scale is responsible for establishing the coarse, overall structure of an image. We therefore provide the embedding (ear). which captures the most abstract, semantic information, as the special Start Token [S] (s)) to initialize the gener- ation. This ensures the whole reconstruction process begins with coherent "Forest" foundation.
5

---

Published as a conference paper at ICLR 2026
Table 1: Quantitative performance comparison on the new NSD test set (Allen et al., 2022). The best and second best results are highlighted in bold and underlined respectively. The Wall-clock inference time for one image reported. t: an auxiliary low-level feature is used.
Reconstruction
Low-Level
High.LLvel
Inference Time
Methods
PixCorr
SSIM
Alex(2) Alex(5) (%)
Ineep (1)
C
SwAV
"BrainDiffuser foy 201
0.273
0.365
90.9
"MindEyel pleatsinn
0.319
0.360
92.8
96.9
94.6
93.3
0.648
0.377
12.29
NeuroPictor
0.375
98.4
94.5
93.3
"MindPye2i
0.322
0:431
96.1
98.6
95.4
93.0
0.619
0.344
12,14
TMindTumen..ooo
0.322
93.8
MindHHier (Ours)
0.32650.0077
0.461=00000
0.461
93.1 +3.1
98.0-004
95.9
95.4.0.50
0.345-02007
2.64
Takagi al. ewwares
0.246
0,410
78.9
65.6
83.8
82.1
0.811
0.504
13.08
MinJBridgauwwss
0.151
0.263
95.5
92.4
94.7
Wills Alinnenwwwwngg
0,271
0.328
95.8
98.0
94.3
94.8
0.649
0.373
MindHier (Ours)
0,381
0.381-0000
94.0...2
98.4
95.9 95.9
96.4...95
0.6066-3.002
0.329
2.64
Refining "Trees" (1 < k < K): The subsequent, higher-resolution scales are responsible for further refinement by adding finer details. At these steps, the model is progressively conditioned on intermediate, detail-focused features, S; @h., through multi-head cross-attention operation. The feature index hg is defined as h1 = M provides fMRI features from early transformer blocks guide the later generation process.
where < K. The design cleverly
In practice, this scale-aware guidance strategy is implemented via a selective attention mask, as illustrated in Fig. 2(b). This mechanism governs the information flow: initial, coarse scales are constrained to attend only to deep, semantic fMRI features, while subsequent, fine-grained scales are directed towards the shallower, detail-oriented ones. To obtain the final image i, the quantized vectors for each generated token map f% are summed approximate continuous feature map, f= - lookup(Z, fss, and decoder 'D then generates the final image î D(f). The entire generative model is trained with a standard Cross Entropy loss to predict the ground-truth token maps at each scale. Ultimately, this pipeline operationalizes our core principle: the image is seeded by coarse representation of the "Forest" and progressively refined with the detailed "Trees", all guided by a representation learned directly from a unified, hierarchical fMRI encoder.
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Dataset. We use the Natural Scenes Dataset (NSD) (Allen et al., 2022), publicly available fMRI dataset containing the brain responses of 8 human subjects viewing naturalistic stimuli from COCO (Lin al., 2014). Following prior work (Takagi & Nishimoto, 2023; Scotti et al., 2024), use these shared 1, 000 images as our test set, averaging the three fMRI repetitions. The training set consists of the remaining 30, 000 non-averaged, single-trial fMRI-image pairs.
Training. Our framework is trained in two-stage process. First, we train a hierarchical fMRI encoder, with a frozen CLIP ViT/L-14 serving as the visual/textual encoder. Second, we freeze the IMRI encoder and train a scale-wise autoregressive model, pretrained by Switti (Voronov et al., 2024). Due to limited space, more implementation details are provided in AppendixE.
Testing. Following a recent work (Scotti et al., 2024). we use the recently updated NSD, which includes 1, 000 testing images. The empirical results of MindHier are obtained on this new test set.
Evaluation Metrics. The evaluation spans multiple metrics: low-level similarity (PixCorr pix- elwise correlation between ground truth and reconstructions, SSIM = structural similarity index metric (Wang et al., 2004), AlexNet (Krizhevsky et al., 2012)); high-level semantic features (Iccep (Szegedy et al., 2015), CLIP (Radford et al., 2021), Eff (Tan et al., 2019), SWAV (Caron et al., 2020)). Most metrics are reported two-way identification accuracy, where higher scores are better (t). Two-way identification accuracy refers to the percent correct if the original image embed- ding is more similar to its paired fMRI embedding than to randomly selected fMRI embedding. Eff and SwAV, however, measure the average correlation distance, where a lower score is superior (L). All results are averaged across subjects 2, 5, and who completed all 40 scanning sessions.

---

Published as a conference paper at ICLR 2026
4.2 QUANTITATIVE RESULTS
Performance Comparison. Table presents the quantitative comparison of reconstructions of MindHier against several fMRI-to-image baselines. Our evaluation includes the strong MindEye series (Scotti et al., 2023; 2024) and other leading methods (Takagi & Nishimoto, 2023; Ozcelik & VanRullen, 2023; Wang et al., 2024; Quan et al., 2024; Bao al., 2025). The results demonstrate that MindHier achieves SOTA on several critical metrics. Notably, MindHier surpasses all com- petitors in semantic fidelity, achieving the top scores for Inception V3 (95.9 (95.9%) CLIP (96.4%) and EV distance. This superiority indicates the ability to more accurately capture the core seman- tic content of visual stimuli. With the addition of the low-level feature used in MindEye2, MindHier! achieves comparable low-level metrics while maintaining SOTA performance on high-level metries.
Efficiency Comparison. Addressing the computational overhead of diffusion models is a key moti- vation for MindHier. We analyze the computational efficiency of MindHier in Table I by comparing its inference time against the leading diffusion- methods. The results reveal that MindHier is substantially efficient, requiring only 2.64 seconds generate a 512 x 512 image, marking 4.67x speed-up over the MindEye2 model. This computational efficiency is attributable to two key de- signs: (i) The hierarchical fMRI encoder generates all neural features in single forward pass. (ii) The coarse-to-fine autoregressive model shifts the majority of computation to lower resolu- tions. This stands in stark contrast diffusion models, which must perform many iterative de- noising steps across the full-resolution space, making our framework fundamentally more efficient.
4.3 QUALITATIVE RESULTS
High-Fidelity Reconstruction. We qualita- tively evaluate visual reconstruction fidelity by comparing MindHier against leading diffusion- based methods in Fig. 3. Our framework demonstrates superior semantic fidelity across diverse range of categories, evident in its reconstruction of both simple and complex scenes. For isolated subjects like the giraffe and the teddy bear, MindHier accurately preserves defining characteristics and posture. While other methods may capture the general con- cept, MindHier often reconstructs more simi- lar pose and shape. For instance, it successfully renders the distinct long neck of the giraffe, feature less accurately captured by competitors. More impressively, MindHier excels at preserv- ing key structural and chromatic properties, as shown by the reproduction of the yellow fire hydrant and the teddy bear. Moreover, Mind-
Seen Image
Minditier
MindEye
(Seott ettal.
20233)
MindEye2
(Sooti et
2004))
MindRridge (Wang el
2024)
Figure 3: Qualitative comparison of IMRI-to-image reconstructions. The stimuli include diverse range of classes, from animals and objects to complex indoor and outdoor scenes. All data shown from Subject 1. Hier accurately recovers the position, orientation, and trajectory of a running train relative to its environment, while other diffusion-based methods struggle produce such a coherent result.
Stable and Consistent Reconstruction. Beyond reconstruction fidelity, a key advantage of our ap- proach is its stability and consistency across mul- tiple generation trials. This stability stems from our model's deterministic nature. Unlike stochas- tic diffusion models that begin with a random Gaus- sian noise seed, our generation process is initiated directly from the encoded fMRI features. As il- lustrated in Fig. 4, MindHier consistently produces virtually similar reconstructions of the teddy bear across four trials, reliably capturing its core features, such as its shape and posture. In stark contrast, a leading diffusion-based method like MindBrridg
Socn Image
Thaal 1
Trsal
T
Trill
Mindilier
MiniB-ióge
(Wang et.l.
2034
Mindeys2
202401
Figure 4: Comparison of reconstruction consis- tency across four trials. MindHier demonstrates more stable and consistent outputs, in contrast to the stochastic diffusion models.
7

---

Published as a conference paper at ICLR 2026
hibits significant variability;; its generations, while semantically related to the "stuffed animal" con- cept, vary wildly in appearance and color across trials. This high level of consistency underscores the robustness of our method, ensuring that the generated image is faithful and repeatable decoding of the specific fMRI signal, rather than one of many possible interpretations.
Faithful Reconstruction. To further evalu- ate the ability of MindHier to faithfully recon- struct visual stimuli with high semantic and spatial accuracy, we perform a brain grounding task across a broad range of objects. Follow- ing the UMBRAE pipeline (Xia et al., 2024b), this evaluation involves feeding MindHier's re- constructed images into pretrained Shikra model (Chen et al., 2023a), which then per- forms localization based on 'known' object la- bels from the original image's COCO anno- tation. As illustrated in Fig. 5, the results demonstrate that MindHier consistently gener- ates outputs that are both semantically coherent and spatially accurate. The model successfully localizes not only salient, large-scale objects (e.g., airplane and giraffe) but also smaller, con- textual items within a larger scene (e.g.. bottle and mouse). Notably. the model successfully reconstructs objects that are partially occluded in the original stimulus (e.g., surfboard and keyboard). These results affirm that MindHier's re- constructions are highly faithful to the original stimuli in both structure and spatial arrangement, showcasing a superior capacity recover critical visual information from brain signals.
Small-Sizze Object
Surbborrd
M duse
Person
Donu
Middle Size Object
Airpla ane
Large- Size Object
Figure 5: Qualitative results of brain grounding. For visualization, a bounding box generated from cach re- construction is overlaid on the original visual stimu- lus, highlighting the precise localization of the target.
4.4 DIAGNOSTIC EXPERIMENT
To deconstruct our framework and validate its principal components, we perform three targeted diag- nostic experiments. Our analysis is designed to quantify the efficacy of three pivotal design choices: (i the impact of hierarchical features, (ii) the specific mapping strategy between fMRI encoder blocks and layers of the frozen CLIP vision encoder, and (iii) the role of the scale-aware guidance strategy. To ensure a fair and controlled comparison, all diagnostic experiments are performed on data from Subject 1 while keeping other hyperparameters unchanged.
Impact of Hierarchical Features. We first investigate the effectiveness of our core idea to obtain hi- erarchical features for guiding the coarse-to-fine to-fine scale-wise autoregressive reconstruction. As shown Table2, our full model, which uses "Hierarchical Feature with our full cascaded supervision (i.e., training with Eq. 1+Eq. 2 for each block)" achieves substantial performance gain compared with the other two baselines. The first bascline, "Single Feature", only leverages the terminal, high-level fMRI feature for guidance, and the second one, "Hierarchical Feature (final supervision)", uses the same hierarchical architecture as MindHier but omits the MSE loss supervision for intermediate blocks. The empirical results in Table suggest that, although "Hierarchical Feature (final supervi- 95.1%-955
sion)" shows a modest improvement over "Single Feature" (e.g., 95.1% - 95.4% in CLIP score), our full framework is more favored (e.g., 95.4% - 97.2% in CLIP score). These results suggest that (i) rich, hierarchical representation that captures both semantic and detailed visual cues is necessary, and (ii) explicit, scale-aware neural guidance is crucial to translating the known representational hierarchy of the pretrained CLIP model to our hierarchical fMRI encoder.
Selection of CLIP Vision Encoder Layers. We next validate the design of our CLIP layer mapping strategy. The results are summarized in Table 3. Our method adopts CLIP ViT-L/14 as the visual backbone. Based on the proposed mapping (gm = 4m)), which maps the fMRI signal to the (12, 16, 20. 24)-th layers, we first derive late-layer mapping variant, 9m 16+2mm which aligns fMRI features with visual feature that encode more semantic information from the deeper layers of the vision encoder (the {18, 20, 22, 24) layers). We then provide alternative, 9m 9m 6m, an earlier- layer mapping to the {6, 12, 18, 24 layers. We observe consistent trade-offs between low-level and high-level fidelity. The early-layer mapping yields marginal improvements in low-level metrics
8

---

Published as a conference paper at ICLR 2026
Table 2: Comparison of different IMRI encoder designs using IMRI data from Subject
Neural Representation
PixCorr
SSIM
Alex(2)
Alex(3)
Incept
CLIPP
Em 4
SwAV
Single Feature
0.188
0.358
89.2%
96.6%
95.7%
95.1%
0.610
0.346
Hicrarchical Feature (final supx.)
0.201
0.363
90.8%
97.0%
95.9%
95.4%
0.608
0.339
Hicrarchical Feature (full supv.)
0.273
0.394
97.0%
99.3%
96.7%
97.2%
0.598
0.321
Table Comparison of different CLIP layer mapping mechanisms using IMRI data from Subject 1.
CLIP VIT-L/14 Layer Mapping
PixCorr
SSIM
Alex(2)7
Alex(5)
Incep 1
CLIP 1
Eff
SwAV
9m 16-+2m
{18, 20, 22, 24)
0.226
0.377
92.3%
98.0%
95.7%
95.4%
0.609
0.335
9m
6m
0.283
0.399
97.6%
99.4%
96.3%
94.8%
0.612
0.325
8 4m
[12. 16, 20, 24)
0.273
0.394
97.09
99.3%
96.7%
97.2%
0,598
0.321
(e.g-, 0.283 vs. 0.273 in PixCorr), while the late-layer mapping degrades overall performance. This indicates that (i) aligning with earlier layers provides more foundational visual features at the cost of high-level semantic accuracy, and (ii) aligning with excessively late layers results in features that are too semantically similar and lack the distinctiveness needed for fine-grained reconstruction. In contrast, our balanced mapping achieves strong high-level similarity (i.e., 97.2 97.2%in CLIP) while maintaining excellent low-level similarity (i.e., 99.3% in Alex(5)). This reveals the effectiveness of correlating fMRI signals with a balanced range of intermedittettofiin CLIP layers.
The Role of the Scale-Aware Guidance. Finally, we validate the design of our scale-aware neural guidance by conduct- ing an experiment that inverts the informa- tion flow. In this variant, the generation is seeded with the first fMRI feature (e1) and progressively guided by coarser semantic features (e2,-. ...,, e.). As summarized in Table 4, this inversion leads to a clear degradation in high-level metrics, e.g., the CLIP score falls from 97.2% to 96.1%, and the SwAV distance worsens from 0.321 to 0.330. The performance gap confirms that a principled coarse-to-fing information flow is critical for achieving high-fidelity results, where global context precedes fine details. At the same time, the model's resilience is note- worthy; the fact that performance degrades but does not collapse entirely highlights the inherent robustness of the scale-wise autoregressive architecture. We attribute this the flexibility of the transformer's attention mechanism, which can partially compensate for the suboptimal guidance. Therefore, while the MindHier framework is remarkably robust, our results confirm that its optimal performance is unlocked through the cognitively inspired scale-aware neural guidance strategy.
Table 4: Ablation study on the scale-aware neural guid- ance. The proposed coarse-to- fine strategy is compared against inverted variant on Subject 1.
Method
Incep
CLIP
Eff J SwAV
MindHier (Coarse-to-Fine)
96.7%
97.2%
0.598
0.321
Inverted (Fine-to-Course)
96.2%
96.1%
0.606
0.330
CONCLUSION
In this work, we identify a primary obstacle to high-quality IMRI-to-image reconstruction: the mis- match between fixed guidance and the dynamic generation process in prevailing diffusion-based" methods. To address this, we introduce MindHier, a framework that reframes this task as coarse- to-fine, scale-wise fMRI-guided autoregressive problem. MindHier maps fMRI signals hierar- chy of neural features that are strategically used to guide a reconstruction process, first establishing global semantics at small scales before progressively refining finer details at larger scales. Empiri- cally, this cognitively aligned framework yields reconstructions with superior semantic consistency and structural faithfulness. Furthermore, MindHier achieves a significant 4.67 acceleration in in- ference speed over leading diffusion-basee methods such as MindEye2. Looking forward, we plan to extend MindHier to cross-subject IMRI-to-image reconstruction via multi-subject pretraining and IMRI-tovvieeo reconstruction by incorporating temporal dynamics. It also comes with new chal- lenges and avenues for future research, in particular enhancing the generation of faithful textures, improving the fidelity of facial features, and exploring resource-efficien schemes. Given the rapid evolution in computer vision and cognitive neuroscience over the past few years, we expect more innovations towards these promising directions.

---

Published as a conference paper at ICLR 2026
6 ACKNOWLEDGMENTS
This research supported by "Pioneer" and "Leading Goose" R&D Program of Zhejiang (No. 2024C01161), Fundamental Research Funds for the Central Universities 226-2025-000057) and the State Key Laboratory of Brain Cognition and Brain-inspired Intelligence Technology (No. SKLBI- K2025004).
REFERENCES
Emily J Allen, Ghislain St-Yves, Yihan Wu, Jesse L Breedlove, Jacob S Prince, Logan Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, et al. A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence. Nat. Neurosci., 25(1):116-126 2022. 2, 21
Yutong Bai, Xinyang Geng, Karttikeya Mangalam, Amir Bar, Alan Yuille, Trevor Darrell, Jitendra Malik, and Alexei A Efros. Sequential modeling enables scalable learning for large vision models. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 22861-22872 2024. 3
Guangyin Bao, Zhang, Zixuan Gong, Jialei Zhou, Wei Fan, Kun Yi, Usman Naseem, Liang Hu, and Duoqian Miao. Wills aligner: Multi-subject collaborative brain visual decoding. In AAAI Conference on Artificial Intelligence, pp. 14194-14202, 2025.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Proc. Advances Neural Inf. Process. Syst., pp. 9912-9924, 2020. 6
Keqin Chen, Zhao Zhang, Weili Zeng, Richong Zhang, Feng Zhu, and Rui Zhao. Shikra: Unleashing multimodal Ilm's referential dialogue magic. arXiv preprint arXiv:2306. /5/95, 2023a. 8
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Iya Sutskever. Generative pretraining from pixels. In Proc. ACM Int. Conf. Mach. Learn., pp. 1691-1703, 2020.
Mu Chen, Liulei Li, Wenguan Wang, and Yi Yang. Diffvsgg: Diffusion-driven online video scene graph generation. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 2217222292,22222
Yongwei Chen, Yushi Lan, Shangchen Zhou, Tengfei Wang, and Xingang Pan. Sar3d: Autore- gressive 3d object generation and understanding via multi-scale 3d vqvae. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 28371-28338 2025b. 3
Zijiao Chen, Jiaxin Qing. Tiange Xiang, Wan Lin Yue, and Juan Helen Zhou Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 22710 22710-22720, 2023b.
Sander Dieleman. Diffusion is spectral autoregression, 2024. URL https://sanderr 2024/09/ ottppp/ttaaaaaaaeeeeeeeaaaaaaaaa aut www.raaraiao.reee.iii.......o.
Patrick Esser, Robin Rombach, and Bjorn Ommer. Taming transformers for high-resolution image synthesis. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 12873-12883, 2021. 1,
Tao Fang, Qian Zheng, and Gang Pan. Alleviating the semantic gap for generalized fmri-to-image im-ge reconstruction. Proc. Advances Neural Inf. Process. Syst., pp. 55096-15107 2023.
James S Gao, Alexander Huth, Mark D Lescroart, and Jack Gallant. Pycortex: an interactive
surface visualizer for fmri. Front. Neuroinform. 9:23, 2015. 17
Zixuan Gong, Qi Zhang, Guangyin Bao, Lei Zhu, Rongtao Xu, Ke Liu, Liang Hu, and Duoqian Miao, Mindtuner: Cross-subject visual decoding with visual fingerprint and semantic correction. In AAAI Conference on Artificial Intelligence, pp. 14247-1423 2025.
Ian J Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. In Proc. Advances Neural Inf. Process. Syst. volume 27, 2014.3 3
10

---

Published as a conference paper at ICLR 2026
Jian Han, Jinlai Liu, Yi Jiang, Bin Yan. Yuqi Zhang, Zehuan Yuan, Bingyue Peng, and Xiaobing Liu. Infinity: Scaling bitwise autoregressive modeling for high-resolution image synthesis. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit. pp. 15733-15744, 2025.
Martin N Hebart, Oliver Contier, Lina Teichmann, Adam H Rockter, Charles Y Zheng, Alexis Kid- der, Anna Corriveau, Maryam Vaziri-Pass Pashi-Pa and Chris Baker. Things-data. multimodal collection of large-scale datasets for investigating object representations in human brain and be- havior, Elife, 12:e82580, 2023. 21
Byeongho Heo, Song Park, Dongyoon Han, and Sangdoo Yun. Rotary position embedding for vision transformer. In Proc. Eur. Conf. Comput. Vis., pp. 289-305, 2024. 19
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. In Proc.
Advances Neural Inf. Process. Syst., pp. 6840-6851, 2020. 1
Tomoyasu Horikawa and Yukiyasu Kamitani. Generic decoding of seen and imagined objects using hierarchical visual features. Nat. Commun., 8(1):15037, 2017. 3
Jingyang Huo, Yikai Wang, Yun Wang, Xuelin Qian, Chong Li, Yanwei Fu, and Jianfeng Feng. Neuropictor: Refining fmri-to-image reconstruction via multi-individual pretraining and multi- level modulation. In Proc. Eur. Conf. Comput. Vis., pp. 56-73, 2024. 1
Tero Karras, Samuli Laine, and Timo Aila. A style-based generator architecture for generative adversarial networks. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 4401-4410, 2019.
Kendrick N Kay, Thomas Naselaris, Ryan Prenger, and Jack Gallant. Identifying natural images from human brain activity. Nature, 452(7185):352-355 2008. 3
Diederik Kingma and Max Welling, Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 2013. 3
Alex Krizhevsky, lla Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. In Proc. Advances Neural Inf. Process. Syst., volume 25, 2012. 6
Dongyang Li, Haoyang Qin, Mingyang Wu, Chen Wei, and Quanying Liu. Brainflora: Uncovering brain concept representation via multimodal neural embeddings. In ACM MM, pp. 5577-5586, 2025a. 22
Haoyu Li, Hao Wu, and Badong Chen. Neuraldiffuser: couroscience-inppiee diffusion guidance for fmri visual reconstruction. IEEE Trans. Image Process., 34:552-505 565, 2025b. 3
Liulei Li, Wenguan Wang, and Yi Yang. Human-object interaction detection collaborated with large relation-drin driven diffusion models. In Proc. Advances Neural Inf. Process. Syst., pp. 23655-23678, 2024.1 1
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Proc. Eur. Conf. Comput, Vis., pp. 740-755, 2014. 6
Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu, Dpm-solver: A fast ode solver for diffusion probabilistic model sampling in around 10 steps. In Proc. Advances Neural Inf. Process. Syst., pp. 5775-5787, 2022.
Yizhuo Lu, Changde Du, Qiongyi Zhou, Dianpeng Wang, and Huiguang He. Minddiffuser: Con- trolled image reconstruction from human brain activity with semantic and structural diffusion. In ACM MM, pp. 5899-5908. 2023. 17
Xiaoxiao Ma, Mohan Zhou. Tao Liang, Yalong Bai, Tiejun Zhao, Biye Li, Huaian Chen, and Yi Jin. Star: Scale-wist wise text-conditioned autoregressive image generation. arXiv preprint arXiv:2406. 10797, 2024.
11

---

Published as a conference paper at ICLR 2026
Yoichi Miyawaki, Hajime Uchida, Okito Yamashita, Masa-aki Sato, Yusuke Morito, Hiroki C Tan- abe, Norihiro Sadato, and Yukiyasu Kamitani. Visual image reconstruction from human brain activity using combination of multiscale local image decoders. Neuron, 60(5):915 9295,,999,,0 2008.
Thomas Naselaris, Ryan Prenger, Kendrick N Kay, Michael Oliver, and Jack L Gallant. Bayesian reconstruction of natural images from human brain activity. Neuron, 63(6):902 9000:992299,,000 2009.
David Navon. Forest before trees: The precedence of global features in visual perception. Cogn. Psychol., 9()):53--383 1977.
Shinji Nishimoto, An T Vu, Thomas Naselaris, Yuval Benjamini, Bin Yu, and Jack L Gallant. Re- constructing visual experiences from brain activity evoked by natural movies. Curr Biol., 21(19): 1641-1646, 2011. 3
Furkan Ozcelik and Rufin VanRullen. Natural scene reconstruction from fmri signals using genera- tive latent diffusion. Sci. Rep., 13(1):15666 2023. 3, 7
Furkan Ozcelik, Bhavin Choksi, Milad Mozafari, Leila Reddy, and Rufin VanRullen. Reconstruction of perceived images from fmri patterns and semantic brain exploration using instance-conditioned gans. In Proc. Int. Jt. Conf. Neural Netw, pp. 1-8, 2022.
Guim Perarna,, Joost van de Weijer, Bogdan Raducanu, and Jose M. Álvarez. Invertible Conditional GANs for image editing. In NIPS Workshop on Adversarial Training, 2016. 3
Ruijie Quan, Wenguan Wang. Zhibo Tian, Fan Ma, and Yi Yang. Psychometry: An omnifit model for image reconstruction from human brain activity. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 233-243, 2024. 3,7
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In Proc. ACM Int. Conf. Mach. Learn., pp. 8748-8763, 2021. 1, 3,
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? In Proc. Advances Neural Inf. Process. Syst., pp. 12116-1, 2021. 4
Aditya Ramesh, Mikhail Pavlov, Gabriel Goh, Scott Gray, Chelsea Voss, Alec Radford, Mark Chen, and Ilya Sutskever. Zero-shot text-to-image generation. In Proc. ACM Int. Conf. Mach. Learn., pp. 8821-8831, 2021. 3
Ziqi Ren, Jie Li, Xuetong Xue, Xin Li, Fan Yang, Zhicheng Jiao, and Xinbo Gao. Reconstruct- ing seen image from brain activity by visually- cognitive representation and adversarial learning. Neurolmage, 228: 1 17602, 2021. 3
Severi Rissanen, Markus Heinonen, and Amn Solin. Generative modelling with inverse heat dissi- pation. In Proc. Int. Conf. Learn. Representations, 2023.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proc. IEEE Conf. Comput. Vis. Pattern Recoguit. pp.
2022. 1
Paul Scotti, Atmadeep Banerjee, Jimmie Goode, Stepan Shabalin, Alex Nguyen, Aidan Dempster, Nathalie Verlinde, Elad Yundler, David Weisberg, Kenneth Norman, al. Reconstructing the mind's eye: fmri-to-image with contrastive learning and diffusion priors. In Proc. Advances Neural Inf. Process. Syst., pp. 24705-24728, 2023. 15, 19
Paul Scotti, Mihir Tripathy, Cesar Kadir Torrico Villanueva, Reese Kneeland, Tong Chen, Ashutosh Narang, Charan Santhirasegaran, Jonathan Xu, Thomas Naselaris, Kenneth A Norman, et al. Mindeye2: Shared-subjer subject models enable Enri-to-image with hour of data. In Proc. ACM Int. Conf. Mach. Learn., 2024. 2, 3, 6, 7, 15, 16
12

---

Published as a conference paper at ICLR 2026
Katja Seeliger, Umut Güglüü Luca Ambrogioni, Yagmur Güglütürk, and Marcel AJ Van Gerven. Generative adversarial networks for reconstructing natural images from brain activity. Neurolm- age, 181:775-785, 2018.
Noam Shazeer. Glu variants improve transformer. arXiv preprint arXiv:2002.05202 2020. 19
Kaixin Shen, Ruijie Quan, Jiaxu Miao, Jun Xiao, and Yi Yang. Tarpro: Targeted protection against malicious image editing. In AAAI Conference on Artificial Intelligence, 2026a.
Kaixin Shen, Ruijie Quan, Linchao Zhu, Dong Zheng, Jun Xiao, and Yi Yang. Audioscenic: Audio- driven video scene editing. Int. J. Comput. Vis., 134(1):12, 2026b. 1
Wensong Song, Hong Jiang, Zongxing Yang, Ruijie Quan, and Yi Yang. Insert anything: Image insertion via in-context editing in dit. In AAAI Conference on Artificial Intelligence, 2026.
Jianlin Su, Murtadha Ahmed, Yu Lu, Shengfeng Pan, Wen Bo, and Yunfeng Liu. Roformer: En- hanced transformer with rotary position embedding. Neurocomputing, 568: 127063, 2024. 19
Jingyuan Sun, Mingxiao Li, Zijiao Chen, Yunhao Zhang, Shaonan Wang, and Marie-Francine Moens, Contrast, attend and diffuse to decode high-resolution images from brain activities. In Proc. Advances Neural Inf. Process. Syst., pp. 22332-12343 2023.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Du- mitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich, Going deeper with convolutions. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 1-9, 2015.
Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 14453-14463, 2023. 6,7
Mingxing Tan, Q Efficientnet Le, et al. Rethinking model scaling for convolutional neural networks. In Proc. ACM Int. Conf. Mach. Learn., volume 15, 2019. 6
Keyu Tian, Yi Jiang, Zehuan Yuan, Bingyue Peng. and Liwei Wang. Visual autoregressive modeling: Scalable image generation via next-scale prediction. In Proc. Advances Neural Inf. Process. Syst., pp. 84839-84866 2024. 2, 5,
Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Proc, Advances Neural Inf. Process. Syst., volume 30, 2017. 3
Anton Voronov, Denis Kuznedelev, Mikhail Khoroshikh, Valentin Khrulkov, and Dmitry Baranchuk. Switti: Designing scale-wise transformers for text-to-image synthesis. arXiv preprint
arXiv:2412.01889, 2024. 3, 6, 19
Jinhong Wang, Jian Liu, Dongqi Tang, Weiqiang Wang, Wentong Li, Danny Chen, Jintai Chen, and Jian Wu. Scalable autoregressive monocular depth estimation. Proc. IEEE Conf. Comput. Vis. 2025.
Pattern Recognit., pp.
Shizun Wang, Songhua Liu, Zhenxiong Tan, and Xinchao Wang. Mindbridge: A cross-subject brain decoding framework. In Proc. IEEE Conf. Comput. Vis. Pattern Recognit., pp. 11333-11342, 2024. 3, 19
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Trans. Image Process., 13(4):600-61) 612, 2004.
Weihao Xia, Raoul De Charette, Cengiz Oztireli, and Jing-Hao Xue. Dream: Visual decoding from reversing human visual system. In Proceedings of the IEEE/CVF Winter Conference Applications of Computer Vision, pp. 8226-8235. 2024a. 3
Weihao Xia, Raoul de Charette, Cengiz Oztireli, and Jing- Hao Xue. Umbrac: Unified multimodal brain decoding. In Proc. Eur. Conf. Comput. Vis., pp. 242-259. 2024b. 8
Jingjing Xu, Xu Sun, Zhiyuan Zhang. Guangxiang Zhao, and Junyang Lin. Understanding and improving layer normalization. In Proc. Advances Neural Inf. Process. Syst., volume 32, 2019. 19
13

---

Published as a conference paper at ICLR 2026
Xingqian Xu, Zhangyang Wang, Gong Zhang, Kai Wang, and Humphrey Shi. Versatile diffusion: Text, images and variations all in one diffusion model. In Proc. IEEE Int. Conf. Comput. Vis., pp. 7754-7765, 2023. 1
Shuchen Xue, Zhaoqiang Liu, Fei Chen, Shifeng Zhang, Tianyang Hu, Enze Xie, and Zhenguo Li. Accelerating diffusion sampling with optimized time steps. In Proc. IEEE Conf. Compul. Vis. Pattern Recognit. pp. 1292-8301. 2024.
Jiahui Yu, Xin Li, Jing Yu Koh, Han Zhang. Ruoming Pang, James Qin, Alexander Ku, Yuanzhong Xu, Jason Baldridge, and Yonghui Wu. Vector-quantized image modeling with improved vqgan. In Proc. Int. Conf. Learn. Reprexentations, 2022.
Bohan Zeng, Shanglin Li, Xuhui Liu, Sicheng Gao, Xiaolong Jiang, Xu Tang, Yao Hu, Jianzhuang Liu, and Baochang Zhang. Controllable mind visual diffusion model. In AAAI Conference Artificial Intelligence, pp. 6935-6943, 2024." 3
Biao Zhang and Rico Sennrich. Root mean square layer normalization. In Proc. Advances Neural Inf. Process. Syst., volume 32, 2019.
Jinzhi Zhang, Feng Xiong, and Mu Xu. 3d representation in 512-hyteevariatiinaal tokenizer the key for autoregressive 3d generation. arXiv preprint arXiv:2412.02202, 2024. 3
Jingyi Zhou, Jiamu Sheng, Peng Ye, Jiayuan Fan, Tong He, Bin Wang, and Tao Chen. Exploring multi-timestep multi-stage diffusion features for hyperspectral image classification. IEEE Trans. Geosci. Remote, Sens., 62: 12:1-16, 2024. 1
14

---

Published as a conference paper at ICLR 2026
A SUMMARY OF THE APPENDIX
We provide additional details in this supplementary material, which are organized as follows:
B elaborates on the data and code availability.
$C presents more experimental results.
DD offers more qualitative results.
SE introduces more implementation details.
SF provides a deeper analysis of inference time.
$G details the visual question answering evaluation.
S displays expanded subject-specific visualizations.
SI shows quantitative results on a new Thing-fMRI dataset.
SJ discloses the use Large Language Model in this work.
B DATA & CODE AVAILABILITY
Data. This work is conducted on the Natural Scenes Dataset (NSD), a large-scale, publicly available IMRI dataset. NSD contains high-resolution, whole-brain fMRI recordings from eight subjects who viewed thousands of colored nature scenes over 30 to 40 scan sessions. The full dataset is accessible upon completion of the NSD Data Access Agreement. Following Scotti et al., 2023; 2024, we use preprocessed flattened fMRI voxels in 1.8-mm native volume space. The data is filtered using unique "nsdgeneral" mask in NSD, which isolates subset of voxels that exhibit the strongest response to visual stimuli within the posterior cortex.
Code. The supplementary material provides the source code for the core components (fMRI encoder and scale-wise neural guidance) of our proposed method. The complete codebase for both training and inference will be made publicly available upon acceptance. We hope that our findings will encourage more innovations in this research field.
Pseudo Code. To guarantee reproducibility, we provide the pseudo codes in PyTorch style for MindHier's key modules, which are given Algorithm and Algorithm2.
Algorithm Pseudo-code for Hierarchical fMRI Encoder of MindHier in a PyTorch-like style.
Models:
EXRI_encoders The EMRI encader with M blecks. ví swal_enceder: Frozan CLTP ViT/1.-14 visual encodar.
# text_encoder: Frozen CLIP viT/I-14 text enooder.
(fmri_signales images, text_captions): Extract featuras from CLTP encoders. multi_ Littiitter_vvsuall..eee Lusts visual,encoder (inageess,, sutput_kiidee sttalddsen_stttserrereeee
text text__ fosto text (tox__ctt
Xap signals enc..ee (fnrr_fffi___ee (fari_sgaaa)))))) X features
aierarchy predicted
frrt__fetsshieearcce
3. Compute alignment icssea between predicted and ground-truth features. loss_mse 0
for range
v_gm 12.normalize haal::i_tiie_yviaaa__eeeee
loss_mse +- MSELoss(e_Ntexc Atext (M), v_gm) a Enforces atructural cozrespondence (Bg. 1)
Align final FMRT feature with both visual and features in shared space.
e_M ses__ffats_hherrrch/.
cext_feats
(t_)text v_gM) SoftCLIPLoss (c_ Atext (M), lass_saftclip SoftCLIPLOss Atext
Rnforces somantic correspondence (Rq. 2)
4. Combine lesses and updase the IMRI encoder.
total_loss less msc 1ass_softclip
total_loss. totai_ioss.backwerrdd) (ackwardd setimizer.step((
return dotal_lost
C MORE EXPERIMENTS
We conduct two additional experiments to demonstrate the advantages of the MindHier framework: strong per-subject performance and its resource-free nature.
15

---

Published as a conference paper at ICLR 2026
Algorithm 2 Pseudo-code for the Scale-Wise Neural Guidance for Image Reconstruction of Mind- Hier in a PyTorch-like style.
Models:
FMRT Lenoodee::: The trafned
encoder,
ar tottnnnonommm: The susle-wise suteregressive transformer.
tokonizers derooooor_ddooorr The decader part of the milti-scale VQ-VXE,
def reconstruct image (Imri_ signiss))
signal using trained encuder, pari_fostures AMRI encoder (fmri (fmri_signal) & Ratch of M. feature tensors --- Stage 1: Seeding the "Forest"
Extract multi-tier features
start_tokenn meen ffw.riffssturrsaafaff
Generate the initial, 1ow-resolution rl_hat ar.transformer (S=start_ttoke)) generated_token_nnee
[x1_hat]
--- Stage 2:
is the Rendering the "Trees" (Progressive, detailed refinement)
number of scales the autoregresaive model. for in range K1:
# Select the appropriate EMRT feature to quide the current scale ktl. feature_idk
floor (M X)
scale cond
mnri_features [feature_idx]
# Predict the next token map conditiened on previous and the fMRI feature. rk het ar ar..rrasfoorer (prenfo generated_toker papp__aasspppppp paps.appanddrrk_N --- Pinal Image Decoding
canditionruesee.com
lookup rstrieves voctors from the codchook &, hat sum (1o r_hat) for r_hat in
reconstrucced__ineg tokenizer dekeoeeer.decoder((( decoder return reconstructed_
Single-Subject Evaluations. Table S1 shows exhaustive evaluation metrics computed for each subject. The empirical results demon- strate the model's strong and consistent perfor- mance across the cohort. While a non-trivial inter-subject variance present, which we tribute to the inherent heterogeneity in the un- derlying data, the model's efficacy promised. Crucially, the reconstruction perfor- mance for every subject significantly surpasses established baselines. This indicates that our model exhibits strong robustness and generalization capabilities across diverse individuals, a critical property for real-world deployment.
Table S1: Quantitative results of each subject.
52
PixCor
0.273
0.237
0,218
0.213
SSIM f
0.394
0.383
0.378
Low
Alex(2)
97.0%
95.1%
91.4%
92.3%
Alex(5)
99.3%
98.8%
97.8%
97.6%
Incep
96.7%
96.1%
97.0%
94.0%
com-
High
CLIP
97.2%
96.0%
97.4%
95.0%
Bff J
0.598
0.611
0.576
0.640
SWAV +
0.321
0.328
0.316
0.352
Finetuning with One Session of Data. Ta- S2 shows that the coarse-to-fine scale-wise autoregressive modeling is resource-efficient. To assess this, we finetune the scale-wise au- toregressive model using single fMRI sion as training data and compare it with the results of MindEye2 (Scotti al., 2024) train- ing with one session of data. We select the two best-performing subjects (S1, S5) on Mind- Eye2 for comparison and find that our results significantly outperform MindEye2 on multiple metrics (e.g., CLIP and Eff scores).
Table S2: Quantitative results with one session of training data. E: the results from MindEye2.
S
S
PixCorr
0.198
0.235
0.166
0.175
SSIM 1
0.382
0.428
0.365
0.405
Low
Alex(2) t
86.9%
88.0%
84.2%
83. 1%
Alex(5)
94.8%
93.3%
93.4%
91.0%
Incep t
93.4%
83.6%
94.2%
84.3%
CLIP
92.2%
80.8%
92.7%
82.5%
High
Eff L SwAV
0.646
0.798
0.644
0.781
0.350
0.459
0.359
0.444
Preliminary Cross-Subject Experiments. Ta- ble S3 presents quantitative results under cross-subject generalization protocol, where pooled dataset
Table $3: Quantitative results for cross-subject generalization.
the model is pretrained and finetuned on a held-out target subject. In the data-rich regime (40-hour finetuning),
Method
Incep
CL.IP
MindEye2 (40 hour)
96.1%
93.3%
0.609
0.338
MindHier hour)
95.7%
95.8%
0.607
MindEye2 hour)
83.5%
80.7%
0.798
0.459
MindHier demonstrates superior transfcrability,
MindHier (1 hour)
75.4%
80.2%
0.860
0.527
achieving CLIP score of 95.8% and surpassing the MindEye2 baseline (93.5%).
This confirms the
16

---

Published as a conference paper at ICLR 2026
Table S4: Brain Grounding Accuracy (IoU) comparison under single-subject setup.
Grounding Methods
Eval Subject
All
Salient
Salient Creatures
Salient Objects
Inconspicuous
IOU
8.5
IOU
a.c
IOU
acc@0..5
IOU
acc
IOU
MindEye
ST
15,34
18.65
23.83
26.96
29.29
31.64
17.88
21.86
4.74
8.28
UMBRAE-SI
13.72
17.56
21.52
25.14
26.00
29.06
16.64
20.88
4.00
8.08
MindHier
15.87
18.69
24.94
26.30
33,00
32.17
16.17
19.90
4.55
8.97
architecture's ' capacity to Iearn robust, transferable representations across individuals. However, in the few-shot regime (1-hour), observe performance dip compared to the baseline, which attribute to overfitting inherent in applying large-scale autoregressive model to sparse data without specialized regularization, marking clear direction for future optimization.
Quantitative Results for Brain Grounding. T4 assesses the spatial fidelity of reconstructions by comparing Brain Grounding Accuracy against established baselines. We benchmark MindHier against MindEye and the specialized grounding method UMBRAE-SI under the same single-subject setup. In addition to metries for all classes denoted 'All', UMBRAE groups the 80 classes COCO according to their prominence into: 'Salient', being the union of 'Salient Creatures' (people and an- imals) and 'Salient Objects' (e.g., car, bed, table), and 'Inconspicuous' (e.g., knife and toothbrush). The results indicate that MindHier consistently outperforms baselines across most categories, partic- ularly in localizing salient creatures. This quantitative evidence corroborates our qualitative findings, suggesting that our hierarchical guidance mechanism facilitates more precise object localization and superior spatial faithfulness compared to methods relying on global or static conditioning.
Single-Shot Best-of-N. Table S5 investi- gates the trade-off between sampling efficiency and reconstruction quality. While our primary evaluation utilizes a Best-of-N (N=4) selec- tion strategy to ensure rigorous comparison, the single-shot (N=l) performance remains re-
Table S5: Comparison of Single-Shor Shggte-Shotteo (N=1) gen- eration versus Best-of-N selection.
Samples
PixCorr
SSIMT
CLIF
SWAV
Time
0.234
0,380
0.92
N=4
0.235
0.381
96.4%
0.329
2.64
markably robust, achieving a CLIP score 95.0% with a sub-second inference time of 0.92s. This indicates that MindHier exhibits high determinism and stability, capable of generating high-fidelity reconstructions without reliance on stochastic over-sampling.
D MORE QUALITATIVE RESULTS
Neuroscience Interpretability. To evaluate the neuroscience interpretability of the Mind- Hier framework, we follow MindDiffusion (Lu et al., 2023) and employ the pycortex library (Gao al., 2015) for brain activity visualization. We utilize 1-2regularized linear regression models to predict IMRI voxel responses from features extracted from different blocks of our proposed fMRI encoder. As mentioned in Appendix B, there exists a partial overlap between voxels selected via manually defined "nsdgeneral" mask and those within anatomically defined visual cortex regions.
To specifically probe the relationship between the model's hierarchy and the brain's visual processing stream, we focus our analysis on this overlapping subset of voxels. Fig. S1 (first row) visualizes the results on a flattened cor- tical surface, where the early visual cortex is demarcated in red and higher-order visual cor- tices are blue. The analysis reveals that dif- ferent model blocks exhibit functional special- ization in predicting brain activity across the vi- sual hierarchy. Specifically, early blocks are more predictive of activity in the early visual cortex, whereas deeper blocks show corre- spondence with activity in higher-order visual regions. This functional gradient demonstrates a clear hierarchical aligmment between the model's successive layers and the progressive stages of the primate ventral visual stream. Intriguingly, non-visual brain regions (depicted in yellow;
Block
Block 4
Figure S1: The importance of different regions in de- coding IMRI features from each transformer block.
17

---

Published as a conference paper at ICLR 2026
Fig. second row) also exhibit similar response profile, suggesting that features from interme- diate model layers may also reflect the importance of other information in brain activity. Although MindHier does not utilize explicit ROI-based supervision, the model spontaneously learns these functional specializations solely from the data within the "nsdgeneral" mask.
Illustration of the Coarse-to-Fine Reconstruction Process. Fig. S2 provides a qualitative illustra- tion of the coarse-to-fine generative process inte- gral to the MindHier framework. This procedure designed in principled alignment with the "For- est before Trees" tenet from cognitive neuroscience, which posits that human perception begins with a global, coarse overview before proceeding to finer details. As depicted, the reconstruction for each stimulus commences not from random noise, but from an initial low-resolution base that establishes the overall semantic context and foundational struc- ture. Through a scale-wise autoregressive modeling paradigm, the framework progressively refines this initial representation by rendering details at succes-
Figure S2: Illustration of the coarse-to-fine sively higher-resolution scales. This refinement reconstruction process of MindHier. fundamentally distinct from the current dominant diffusion- based methods that typically recover an image from unstructured Gaussian noise through iterative denoising process.
Comparison of Reconstructions Across Sub- jects. Fig.S3 S3 provides qualitative comparison of image reconstructions generated from the fMRI data of four different subjects (S1, $2, S5, and S7). The results demonstrate high degree of consistency in reconstruction quality across these individuals, indicating that the model ro- bustly captures the salient semantic and struc- tural features of the visual input. For instance, across all four subjects, the model successfully reconstructs the defining characteristics of di- verse objects, such as the octagonal shape a stop sign, the stripes of zebra, and the form of an airplane. Although minor inter-subject vari- ability is present in the finer details, which is expected given the inherent differences in neu- ral activity patterns, the core semantic content is consistently and faithfully preserved. This suggests that the proposed framework effectively handles different individuals, decoding shared visual information from distinct brain signals.
GT
SI
S2
S5
S7
STOP
STOP
STOP
STOP
OP
BAKG KV
Figure S3: Qualitative comparison of reconstruc- tions from four different subjects (S1, S2, S5, S7).
Speed vs. Quality. To verify that our reported speedup is result of fundamental architectural efficiency rather than a simple reduction gen- eration steps, conduct controlled ablation study. In our primary evaluation, we utilize the official recommended settings for Mind- Eye2 (38 denoising steps) to ensure it performs its optimal reconstruction quality. Here, we forcibly restrict the diffusion baseline to 10 de- noising steps match the 10 autoregressive scales used by MindHier, thereby aligning the number of generation passes. As illustrated in Fig. S4, this constraint exposes critical limita- tion diffusion- decoding. While Mind- Eye2 produces high-quality results at 38 steps,
Seen Image
MindHier (10 stepel
MindPyy22 C3B steps)
MindRye2 (10 stage)
Figure $4: Qualitative comparisons within stan- dard and equalized generation steps.
18

---

Published as a conference paper at ICLR 2026
reducing the step count causes severe degradation in semantic fidelity. For instance, the fire hydrant (Column 5) loses its structural integrity in the 10-step MindEye2 output, hallucinating a distorted, clown-like figure rather than distinct object. Similarly, the teddy bear (Column 3) suffers from a loss of identity, resulting in a generic, less coherent animal form. This comparison demonstrates that MindHier's efficiency is not a trade-off between quality and speed; rather, it represents more effective utilization of computational steps.
E MORE IMPLEMENTATION DETAILS
The MindHier frameworks follow a standard two-stage training pipeline by first training a hierar- chical (MRI encoder, and then finetuning scale-wise AR with the frozen fMRI encoder.
For optimization of the hierarchical fMRI encoder, we employ A with betas of (0.9, 0.999) and mini-batch size of 512. We adopt cosine annealing policy with a warmup period to schedule the learning rate, which is initialized to x 10-1 The model is trained for a total of 300 epochs.
Regarding the alignment mechanics, the mapping is established between the fMRI feature maps (shape: batch, seq,enn dim) and the corresponding full patch token outputs from CLIP ViT-L/14. To ensure robust optimization, both sets of features undergo L2-normalization prior the computation of the Mean Squared Error (MSE) loss. For the loss configuration, the total objective is a weighted sum of the MSE loss and the SoftCLIP loss. To ensure training stability, we assign a higher weight to the MSE term (2 x 10²) to balance the loss terms, as it presents more challenging optimization landscape. The SoftCLIP temperature parameter
set to a fixed value of 0.005.
The architecture of the scale-wise autoregressive model in MindHier incorporates modern designs of Switti (Voronov al., 2024). The model consists of 30 non-causal transformer layers, each of which is mainly composed of a multi-head self-attention block, a multi-head coss-attention block, and feed-forward network employing the SwiGLU activation function (Shazeer, 2020). For enhanced training stability and efficiency, the model applies RMSNNrm (Zhang & Sennrich, 2019) before and after the attention and FFN blocks. It is also employed for query-key (QK) normalization within the attention mechanism. To accommodate variable input resolutions, the model incorporates RoPE et al., 2024; Heo et al., 2024) as positional encoding. Finally, the fMRI embedding is integrated to condition the model via AdaLN (Xu et al., 2019). Second, train a scale-wise autoregressive model, pretrained by Switti (Voronov et al., 2024). We use the Adam W (3) = 0.9, 82 = 0.95) for optimizer and initialize the learning rate with x 10-4, which is also scheduled via cosine annealing with warmup. Training images are augmented using random crops of 512 x 512 resolution. There are 10 progressive scales for the autoregressive model. At this stage, we use a mini-bate size of 80 and train for 9K iterations. During inference, we follow MindBridge (Wang et al., 2024) and MindEye (Scotti et al., 2023) to reconstruct candidates of each sample for selecting the best one based on CLIP similarity.
freeze the fMRI encoder and
All inference tasks are conducted on single NVIDIA RTX 4090 GPU. Unlike prior state-of-the-art methods, MindHier does not require auxiliary retrieval submodules or separate captioning networks, relying solely on the unified encoder and autoregressive generator.
F A DEEPER ANALYSIS OF INFERENCE TIME
In this section, we conduct two experiments to analyze the efficiency gains of the MindHier model. Component-wise Efficiency Analysis. To in- vestigate the souree of our efficiency gains, we analyze the stage-wise inference latency com- pared to the diffusion-based baseline, Mind- Eye2. As summarized in Table S6, the total in- ference process is dominated by the reconstruc- tion phase. MindHier achieves significant speed up through structural advantages:
Table S6: Quantitative results for visual question answering evaluation.
Model
Encoding
Reconstruction
Refinement
MindEye2 MindHier
0.785s
7.8648
3.323s
0.008s
2.295s
N/A
Elimination of Refinement Stage. Unlike diffusion- based methods such as MindEye2, which rely a separate, computationally expensive refinement stage (3.32s) enhance image quality, MindHier generates high-fidelity outputs in a single autoregressive pass. This structural difference alone removes major computational bottlneckk, eliminating the refinement time.
19

---

Published as a conference paper at ICLR 2026
Table S7: Stage-wise inference time breakdown across autoregressive scales. The latency correlates with image resolution and the majority of inference time is spent on the final high-resolution scales.
Stage
0 (Coarse)
2
3
4
5
6
7
8
(Fine)
Resolution
16 0.028
32
48
64
96
144
202
288
384
512
Time (s)
0.031
0.038
0.040
0.074
0.216
0.126
0.283
0.474
0.985
Efficient Encoding and Reconstruction. Beyond the refinement stage, MindHier demonstrates superior efficiency in the core encoding and reconstruction steps. The encoder operates faster than the baseline (0.008s vs. 0.785s). Furthermore, the reconstruction phase is accelerated (2.295s vs. 7.864s) by employing discrete, scale-wise paradigm rather than iterative diffusion sampling.
Scale-wise Latency Breakdown. We further dissect the reconstruction time across different res- olutions in Table S7. The results reveal that the computational cost is non-uniform and heavily back-loaded. The coarse stages (Stages 0-4), which establish the semantic structure of the image, executed with negligible latency (under 0.1s combined). The majority of the inference time is incurred during the final high-resolution upscaling (Stage 9, 0.985s), where the model resolves fine-graine grae-graindd pixel details. This confirms that while pixel-Ievel generation 512 x 512 resolution re- mains the most demanding step, the hierarchical approach efficiently offloads the semantic planning lower, faster resolutions.
Figure S5: Qualitative results of visual question answering.
G VISUAL QUESTION ANSWERING EVALUATION
To assess faithfulness beyond object detection, extend our evaluation to Visual Ques- tion Answering (VQA) task. Utilizing the pre- trained BLIP-2-OPT-2.7B model, we interro- gate the reconstructed images to determine if the semantic content preserved by the brain de-
Table S8: Quantitative results for visual question answering evaluation.
GroundTruth MindHier
MindEye2
Accuracy
58.68%
43.29%
41.71%
coding pipeline is sufficient for a Vision Vision-Language Model (VLM) to interpret correctly.
Quantitative Results. As shown in Table we measure the top-3 accuracy of the VQA model's responses to the reconstructed images against the ground truth answers. The original Ground Truth (GT) images yield a VQA accuracy of 58.68%, establishing the upper bound for the model's reason- ing capabilities. MindHier achieves accuracy of 43.29%, surpassing the previous state-of the-an art MindEye2 (41.71%). This performance gap indicates that MindHier is more effective at faithfully decoding and preserving the distinct semantic features.
20

---

Published as a conference paper at ICLR 2026
Qualitative Results. Fig. S5 reveals that MindHier excels at retaining the "gist" and categorical identity of the scene, though fine aeee-grined discrepancies remain:
Semantic Retention: In distinct scenarios, such the fruit display (ID: 334208) and the zebras (ID: 491611), the reconstructions successfully trigger correct VQA responses regarding object presence ("fruits and vegetables"), attributes ("stripes"), and location ("Africa").. This suggests high fidelity in faithfully decoding dominant subjects.
Texture and Material Ambiguity: While object classes are generally preserved, specific material properties can be altered during reconstruction. For instance, in the tower scene (ID: 3231 182), the VQA model correctly identifies the "clock," " but misidentifies the building material as "brick" rather than "stone" in the reconstruction, pointing to a loss of eigh-frequency textural information. Fine-Grained Counting Limitations: The evaluation also highlights challenges with precise counting and small details. In the skiing example (ID): 21 21826), while the action ("skiing") is cor- rectly classified, the reconstruction fails to preserve the exact number of ski poles held. Similarly, in the harbor scene (ID: 570543), the reconstruction distorts the vessel enough that the model fails to count "one" sailboat, despite correctly identifying the context as a harbor containing boat.
Overall, while pixel-perfect reconstruction remains challenge for fMRI decoding, MindHier demonstrates a superior capacity for faithfully reconstructing brain signals into semantically co- herent visuals that align closer to the ground truth interpretation than competing methods.
H EXPANDED SUBJECT-SPECINE VISUALIZATIONS
To provide a comprehensive qualitative assessment of MindHier, we present an expanded gallery of reconstructions of Subj 5. and in S6-SS, This diverse collection covers a wide spec- trum of semantic categories-including wildlife, transportation, food, sports, and indoor environ- ments demonstrating the model's ability generalize across varied visual stimuli.
Semantic Consistency and Object Identity. The results highlight MindHier's strength in semantic grounding. For distinct, salience-heavy objects, the reconstructions are remarkably accurate.
Animals: In images of animals (e.g., zebras, giraffes, bears), the model successfully captures not
only the species but also the pose and context (e.g., bear water, a zebra grazing).
Transportation: Rigid structures like airplanes, trains, and buses are reconstructed with correct orientation and scale relative to the background.
Food: Culinary items like pizza and cakes are generated with appetizing fidelity, preserving the general category and color palette, though specific toppings may vary.
Scene Layout and Atmosphere. Beyond single objects, the model shows competence in scene derstanding. In complex indoor environments (e.g.. bedrooms, bathrooms), the spatial arrangement of furniture, such as the placement of bed relative to window, is generally preserved. Similarly, the global lighting conditions and color histograms (e.g., the blue hues of a surfing scene vs. the warm tones of dining room) consistently matched, indicating effective decoding of low-level visual features alongside high-level semantics.
Limitations and Failure Cases. Despite these successes, a closer inspection reveals characteristic failure modes common in fMRI decoding:
Fine-Grained Details: While the "gist" is preserved, specific high-frequency details often hallu- cinate. For example, while "pizza" is correctly reconstructed, the exact arrangement of pepperoni slices differs from the ground truth.
Text and Symbols: The model struggles to reconstruct legible text or specific logos (e.g., on signage or vehicles), treating them instead as generic texture patterns.
Human Identity: While the presence and posture of humans are detected, facial features remain generic or blurred, lacking the precision required for identity recognition.
GENERALIZABILITY ON THE THINGS DATASET
To demonstrate the robustness and generalizability of MindHier beyond the NSD benchmark et al., 2022), we extend our evaluation to the THINGS-NN -MMI dataset (Hebart et al., 2023). This dataset presents challenging scenario due to its distinct acquisition protocols compared NSD.
21

---

Published as a conference paper at ICLR 2026
Table S9: Quantitative results on THINGS-IMRI set. The best results are highlighted in bold.
Method
PixCorr
SSIM
Alex(2)
Alex(5) f
Incep 1
CLIPT
Eff
SwAV
BrainFlora Minnus)
0.079
0.348
0.595
62.2%
60.4%
57.6%
0.812
0.661
MindHHier (Ours)
0.109
0.357
0.710
81.9%
71.9%
73.1%
0.900
0.589
Experimental Setup. The THINGS-IMER dataset includes fMRI recordings from three subjects viewing 720 concepts (12 images per concept) during training and 100 concepts (1 image per con- cept) during testing. To ensure rigorous and fair comparison with the state-of-the-art method this benchmark, BrainFlora et al., 2025a), we adopt their exact experimental setting.
Specifically, we employ a joint-subject training strategy where fMRI voxels from all three subjects are padded to a unified dimension (7000). This padding strategy accommodates the varying voxel counts across subjects, allowing hierarchical encoder to be trained simultaneously data from all three individuals. We use the same hyperparameters described in the main text ($4.1), adapted only for the input dimension changes required by the THINGS dataset.
Quantitative Analysis. Table S9 presents the quantitative comparison between MindHier and the baseline BrainFlora on the THINGS dataset. The results demonstrate that our hierarchical autore- gressive approach generalizes effectively to new data distributions and multi multi-subject settings.
Most notably, MindHier exhibits substantial improvements high-lewe level semantic retrieval metries. We observe 19.7% increase in AlexNet(S) accuracy (62.2% - 81.9%) and a 15.5% increase in CLIP identification accuracy (57.6% - 73.1% 73) compared the baseline. These gains confirm that our Scale-Aware Coarse-to-Fine Neural Guidance successfully captures and generates the semantic "gist" of diverse concepts found in the THINGS dataset. Furthermore, our method improves low- level structural fidelity, evidenced by higher PixCorr (0.079 + 0.109) and SSIM (0.348 0.357) scores. The reduction in AwAV distance (0.661 - 0.589) further indicates that our generated images perceptually aligned with the ground truth in deep feature spaces.
J THE USE OF LARGE LANGUAGE MODEL
In the preparation of this manuscript, Large Language Model (LLM) was employed writing assistant to aid in polishing the text and improving clarity. The authors fully developed all concepts, arguments, and conclusions presented. Following the LLM'' assistance, every portion of the text was meticulously reviewed, fact-checked, and rewritten by the authors to ensure the final content accurately and entirely reflects our own work and intended meaning.
22

---

Published as a conference paper at ICLR 2026
Ours
GT
Ours
GT
Ours
GT
Ours
GT
Ours
GT
Ours
Figure S6: Visualizations of MMRI-to-image reconstruction for Subject-1.
23

---

Published as a conference paper at ICLR 2026
Ours
Ours
GT
Ours
GT
Ours
GT
Ours
Ours
Figure S7: Visualizations of fMRI-to-image reconstruction for Subject-2.
24

---

Published as a conference paper at ICLR 2026
Ours
Ours
GT
Ours
GT
Ours
GT
Ours
Ours
Figure S8: Visualizations of fMRI-to-image reconstruction for Subject-5.
25

---

Published as a conference paper at ICLR 2026
GT
Ours
GT
Ours
GT
Ours
GT
Ours
GT
Ours
GT
Ours
Figure 89: Visualizations of fMRI-to-image reconstruction for Subject-7.
26


Reviews:
[{"review_id": "hVB14Hc2YR", "reviewer": "Reviewer_YmRz", "summary": "This paper focuses on the task of fMRI-to-image decoding. Inspired by the hierarchical processing mechanism of the human visual system, the authors designed a corresponding model. During the fMRI representation learning stage, the authors aligned the intermediate features of the fMRI representation model with the CLIP image features at each stage. In the reconstruction stage, they introduced the paradigm of a VAR model to progressively reconstruct images from coarse to fine scales. I believe this process effectively simulates the hierarchical processing mechanism of the human visual system. Moreover, the proposed method achieves better reconstruction quality and efficiency. The experiments and evaluations in the paper are well-conducted, though I think some additional results would be beneficial.", "questions": "1. In Table 1, the authors compare several multi-subject/cross-subject methods, such as MindEye2, MindBridge, and Wills Aligner. What I am uncertain about is whether their evaluation was conducted under a multi-subject training setup. If the proposed method is still single-subject (i.e., training a separate model for each subject), I think this could be considered a limitation.\n\n2. In lines 329–330, the authors mention that using the blurred images from MindEye2 would significantly improve the low-level metrics. Could the authors provide the corresponding results to support this claim?\n\n3. In the “Faithful Reconstruction” section, the authors use an object detection task to evaluate the fidelity or reliability of the reconstructed images. I wonder whether directly using the VQA task from MSCOCO would be a better choice? By employing a well-established multimodal model, one could compare the VQA accuracy between the reconstructed images and the ground truth images.\n\n4. The authors mention the MindTuner method in the references, but it is not included in the experimental comparisons. I wonder why this method was not considered in the experiments.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "+ The motivation of this paper is clear, and the proposed method effectively addresses it.\n\n+ The manuscript of this paper is well-structured.\n\n+ The experimental results achieved in this paper are quite good.", "weaknesses": "+ I believe that the method in this paper does not fully simulate the hierarchical processing mechanism of the human visual system during the fMRI representation learning stage. Specifically, different brain regions play distinct roles at various stages of the hierarchical processing mechanism. Therefore, I think dividing the fMRI signals into multiple brain regions, extracting representations separately, and then integrating them would better align with the biological mechanism. This approach might reduce accuracy, but I believe such an attempt could still be meaningful.\n\n+ The improvements in both performance and inference speed reported in the paper are likely due to the use of a more advanced generative model — specifically, replacing SDXL with the scale-wise VAR model.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "IYGNYRqSoH", "reviewer": "Reviewer_Pfeq", "summary": "The paper presents a well-motivated and technically sound alternative to diffusion-based fMRI-to-image reconstruction. The proposed hierarchical alignment and scale-aware autoregressive generation are novel and effectively address the limitations of static guidance in existing diffusion-based methods. The results are strong, especially in inference speed.", "questions": "Please see the Weaknesses section.", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This work has a well-motivated approach. The idea of aligning hierarchical fMRI features with multi-scale image generation is innovative and biologically plausible, echoing the \"forest before trees\" principle in human perception.\n\n2. The use of a scale-wise autoregressive model (VAR) is a fresh direction compared to the overused diffusion models. And the VAR-based method achieves competitive results on multiple high-level metrics with a faster speed. Besides that, the reconstruction results are stable and repeatable.\n\n3. The article is well-written and very easy to understand.", "weaknesses": "1. I noticed that text information was also used in the model's input. It is necessary to provide another result that uses only text information as the input. This is how we can determine whether the model is translating the fMRI data or is more dependent on the text information. Because the pre-trained VAR is a text-to-image generation model, there is concern that fMRI does not play a major role in this model.\n\n2. In terms of speed, I understand that most of the steps in the VAR model are carried out on a small scale. However, it would be better for the article to provide the number of inference steps for the corresponding comparison diffusion models, or those comparison models might not require so many inference steps (for example, 50 steps). This way, the quality can be compared at the same speed.\n\n3. The training process is somewhat complex. The two-stage training (fMRI encoder + autoregressive model) and the need for hierarchical alignment may increase implementation complexity and hyperparameter sensitivity.\n\n4. Although single-subject results are strong, the paper does not deeply explore cross-subject generalization or model adaptation to new subjects with limited data. In current research, there are very few studies that focus solely on a single subject. It is preferable to include results from multiple subjects or unseen subjects, as this is crucial for practical applications.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aY7REfqG70", "reviewer": "Reviewer_DrWf", "summary": "The paper introduces a new pipeline for reconstructing images from fMRI beta-values on the NSD dataset. Instead of relying on the rather common approach of decoding a single visual embedding from fMRI (e.g. CLIP), which conditions a diffusion-based image-generation model (e.g. via U-Net cross-attention layers), the paper uses a multi-scale architecture that simultaneously aligns various layers of a CLIP visual encoder to the successive blocks of a transformer fMRI encoder (with the output transformer block being aligned both with the output layers of the visual encoder and of an additional text encoder). Then, the activations from these aligned blocks are used to condition an autoregressive image generator based on the Switti model (Voronov et al, 2024), to inform the reconstruction process both at coarse- and detailed-level. The paper shows superior reconstruction performance against recent fMRI-to-Image baselines on the NSD dataset on semantic metrics, 4x inference speedup and more stable / deterministic reconstructions given the same fMRI sample.", "questions": "- Typo L316: 'Two-way identification refers to percent correct if ...'\n- In Table 1, it is unclear how the the auxiliary low-level image is used for conditioning the image generation of the autoregressive model to gain competitive performance against baselines using a low-level-feature (E.g. MindEye2)\n- It is not easy to understand exactly how the outputs of the transformer blocks condition the autoregressive generation via cross-attention. There is a mention of AdaLN but the expected fMRI-decoding audience may not be able to grasp the details of this procedure without further details. In particular, are the weights of the autoregressive model fine-tuned during the process or left frozen ?\n- It is not explicitly stated whether the 'text encoder output' / 'visual encoder output' and 'fMRI encoder output' have the same dimensionality, allowing for the Eq1+Eq2 loss. For example, most CLIP implementations have a different output shape for text and image, how are these reconciled to be both trained against the last fMRI encoder's block output ? \n- The description of f_k in the scale-aware guidance is a bit difficult to follow. My understanding of f1 is clear, but I'm not sure what f_k is: it is described as 'residuals', but it is unclear that all 2D token maps have the same shape (because of the notation h_k x w_k for their size, which seems to depend on k). If these 2D maps have different dimensions depending on k, how are residuals defined ?\n- I'm not sure how the claim of MindEye2 being 'highly optimized' is supported. It is also unclear whether the inference-time gains are mostly due to the diffusion model from ME2 being slow (i.e. SDXL), or if the bottleneck is the inference of ME2's brain-to-CLIP module itself. A more detailed analysis of where the inference time is spent in each pipeline and which components from MindHier / ME2 actually contribute to the claimed 4x factor would increase clarity (e.g. time spent image generation model vs fMRI encoding module). \n- Some recent fMRI-to-Image baselines comparable with ME2 are missing (e.g. NeuroPictor)\n- There are various known issues with the NSD dataset (e.g. poor categorical diversity, see Kamitani et al). The impact of the paper could be increased by including an analysis on an additional dataset.\n- How is the multi-scale conditioning strategy tied to autoregressive nature  of the image generation model ? Couldn't a diffusion-based model be informed similarly of various scales (at various denoising steps) ?", "limitations": "", "rating": 2, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper is easy to follow and the experiments are detailed\n- The use of an autoregressive generation model for image reconstruction from fMRI has been largely unexplored and the paper is a rather welcomed initiative in the field\n- The reconstructions shown are impressive and the stability of reconstructions is a rather attractive feature for potential future BCI applications\n- Important ablations on the hierarchical module are presented and lead to interesting conclusions (e.g. earlier features providing increasing low-level metrics and later layers globally decreasing performance)", "weaknesses": "- A number of claims are rather strong compared to the results supporting them. For example, at L339 it is claimed that the author's framework is 'fundamentally more efficient' than ME2 but it is not clear what set of results supports the 'fundamental' aspect of this claim: the inference time bottleneck for each pipeline is not detailed, and thus it is unclear if this performance gain in inference-time is a property of the fMRI-to-Image pipelines or linked to the efficiency of the diffusion vs autoregressive image generation models at hand.\n- The paper reports superior performance on image similarity metrics, which is great. However these metrics are known to be unreflective of qualitative / human evaluation and the paper only compares a very small set (5 images) of reconstructed images to baselines, which is insufficient to support the claims from Section 4.3 (the reconstructed images are arguably very close to the ones from ME2 and the distinctive characteristics of MindHier are supported only by two of these images).  These claims would be better supported by showing a larger set reconstructions, and a more representative set of success and failure modes for this task. \n- Similarly, the consistency of multiple generations is compared only to a single baseline, which undermines the claim of superior performance for MindHier. Again, this claim would be better supported by showing more images, and more baselines.\n- The same comment holds for the bounding box analysis (from UMBRAE), which does not show the same results for baselines, and thus makes it impossible to assess whether MindHier is actually an improvement.\n- The repeated statement that previous approaches rely on the decoding of a single high-level embedding is incorrect and misleading, as several previous methods e.g. Brain Captioning (Ferrante et al), MindEye (Scotti et al) use additional low-level data from the stimuli to condition the reconstruction of images (i.e. Depth Maps or VAE latents)\n- The approach is applied only on the NSD dataset and its conclusions could be better supported if the analysis included another dataset (e.g. THINGS-Image, BOLD5K, ...)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KXFu8fWqr8", "reviewer": "Reviewer_u1wj", "summary": "The paper replaces diffusion with a scale-aware autoregressive (AR) generator conditioned on a hierarchical fMRI encoder that aligns to intermediate CLIP features. The idea is “forest-before-trees”: coarse scales use deeper, high-level brain features; finer scales use shallower ones. On NSD, the method reports good high-level identification and faster inference than diffusion baselines.\n\nOverall: promising direction, but several method details are under-specified and the experimental comparisons aren’t yet on fully solid ground.", "questions": "What I’d like to see in the rebuttal (specific and actionable)\n\n- Define the “† low-level feature” and how it is injected; either give baselines an equivalent cue or drop the “†” rows.\n\n- AR training/inference details: causal masking, teacher forcing, decoding strategy; also report single-shot results (no best-of-N) and, if using selection, ensure it only references fMRI-derived embeddings. Loss/temperature/weighting hyper-params and a short sensitivity analysis.\n\n- Ablations: K/M sweeps; learned routing vs. fixed schedule; start-token variants; with/without contrastive term; with/without the “†” cue.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- Autoregression instead of diffusion: Clear potential for speedups, simpler likelihood training, and more direct optimization of image tokens (rather than only pushing CLIP embeddings).\n\n- Hierarchy-to-hierarchy conditioning: Aligning fMRI to multiple CLIP layers—and using those layers across scales—fits the coarse→fine generation narrative and is technically interesting.\n\n- Semantic metrics look strong: The method seems particularly good at high-level, identity/semantic recognition, which is where most practical interest is today.", "weaknesses": "1. Under-specified alignment mechanics. The paper does not clearly explain how fMRI features are matched to CLIP layer outputs: token vs CLS usage, pooling, projection dimensions, normalization, or the exact shapes at each level. This affects both reproducibility and interpretation of the model. Similarly, loss weighting and SoftCLIP details are missing. Multiple MSE/contrastive terms are simply summed, but there’s no report of weighting, temperature, negative sampling, or sensitivity analyses. Given that evaluation leans on CLIP-style metrics, this raises target–metric coupling concerns.\n\n2. Start-token information bottleneck. The description suggests collapsing the deepest fMRI feature into a single start token. That risks throwing away critical information right at the first coarse scale. A learnable projection of the full feature (or multiple tokens) would be more principled.\n\n3. Hand-crafted scale↔hierarchy schedule. The mapping from image scale to CLIP/fMRI layer is fixed rather than learned. When K (scales) and M (feature levels) don’t match, some levels repeat or are skipped. A learned router/attention over {e_m} per scale would be more robust.\n\n4. Conditioning path is muddy. It’s unclear when the model uses cross-attention vs. AdaLN/FiLM-style conditioning vs. a “selective attention mask,” and where each sits in the block. This matters for understanding how and when brain information actually influences generation.\n\n5. AR claim vs. non-causal blocks. If the backbone uses non-causal transformers, how is the AR factorization enforced (masking, teacher forcing, decoding strategy)? Please make the training/inference story self-consistent.\n\n6. “Deterministic” vs. best-of-N selection. If multiple candidates are generated and the best is picked by CLIP similarity, claims about determinism/consistency need to be backed by single-shot results and by selection that only uses fMRI-derived embeddings (not ground truth).\n\n7. Ablations are thin and subject-limited. Key ablations (K/M sweeps, learned routing vs. fixed schedule, removing the contrastive term, different start-token strategies, with/without “†” low-level cue) should be run and reported, preferably beyond a single subject.\n\n8. Interpretability figures look off and lack statistical grounding. The brain maps appear asymmetric and not obviously aligned with canonical visual cortex topology. More importantly, if the encoder was trained to mimic CLIP hierarchy, qualitative maps can be circular. Please add ROI-wise linear encoding with cross-validated R² and noise ceilings, and run RSA/CKA against both brain RDMs and CLIP to show added brain-specific structure rather than merely recapitulating CLIP.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "tMpn941V8C", "reviewer": "Reviewer_3r2V", "summary": "This paper proposes MindHier, an fMRI‑to‑image reconstruction framework that replaces the diffusion-based pipeline with a autoregressive one. The method (i) learns a Hierarchical fMRI Encoder whose intermediate blocks are aligned layer‑by‑layer to CLIP’s vision backbone, and injects these fMRI features as scale‑aware guidance during generation. On NSD, the approach achieves compelling performance with faster inference than diffusion-based method.", "questions": "- In Fig. S1, the paper states that EVC in red and higher‑order visual cortices in blue, but the overlays do not obviously track standard visual regions. Could you clarify?\n- In Fig. 4. if the model is deterministic, why do we see non‑identical repetitions in some examples? Which parts of the pipeline are stochastic?\n- Please detail the hardware and settings used to report 2.64 s and the 4.67× speedup.\n- In Table 4 column header, “Eff ↑”, but Eff is a distance and should be ↓?", "limitations": "", "rating": 4, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The paper is, to my knowledge, the first to systematically test a scale‑wise AR pipeline for fMRI reconstruction, rather than diffusion. The coarse‑to‑fine conditioning matches the generative schedule of AR, and the hierarchy‑to‑hierarchy alignment is cleanly specified\n- Achieves competitive performance with SOTA while running in 2.64 s per image. The paper explains the speedups via a single forward pass and concentrated computation at low resolutions in AR.\n- Ablation experiments quantify (i) the value of hierarchical supervision across blocks, (ii) CLIP‑layer mapping trade‑offs , and (iii) the benefit of coarse to fine guidance.\n- Fig. 4 shows lower trial‑to‑trial variability than a diffusion baseline, consistent with the deterministic AR start from fMRI features . (But see my questions below.) \n- Data split details, PyTorch‑style pseudo‑code, and a code‑release commitment are provided", "weaknesses": "- While I acknowledge the value of validating the method on a different architecture, although it achieves the best score on several metrics, there are multiple cases where the variances suggest marginal improvement or almost no improvement over prior work. Although the emphasis on inference speed is understandable, the quantitative gains appear limited overall.\n- The paper is primarily an engineering contribution; deeper neuro discussion would strengthen it. In Fig. S1, the paper states that EVC in red and higher‑order visual cortices in blue, but the overlays do not obviously track standard visual regions.\n- Although §4.3 is explicitly labeled “qualitative” and therefore does not include quantitative evaluation, showing a handful of examples is not sufficient to substantiate the claims. It remains unclear what the method enables that prior approaches could not. If the authors wish to advance these claims, please quantify each sub‑claim and then provide qualitative interpretation on top of the numbers.\n- I could not find explicit specifications for the HFE depth, the number of AR scales, or the codebook size, nor ablations over these choices.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bqNeFGG2oy", "reviewer": "Reviewer_mNCs", "summary": "This paper presents MindHier, a coarse-to-fine framework for fMRI-to-image reconstruction based on scale-wise autoregressive modeling. Unlike diffusion models that rely on a fixed global embedding, MindHier captures multi-level neural information through a hierarchical fMRI encoder, aligns fMRI and CLIP features layer by layer, and injects embeddings at matching scales for progressive reconstruction. Experiments on the NSD dataset demonstrate improved semantic fidelity, faster inference, and more stable outputs compared to diffusion-based baselines.", "questions": "* How does MindHier fundamentally differ from adaptive-guidance approaches like Neuropictor or DecoFuse beyond adopting autoregression?\n* Could you clarify the purpose of each block in Fig. 2(b) and the rationale behind the hierarchy-to-hierarchy mapping?\n* What is the expected real-world impact of reducing inference time for fMRI decoding tasks?\n* How significant are the reported improvements statistically, and do they justify the added model complexity?\n* Could you quantify how much each proposed module contributes to the performance improvement?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "* The hierarchy-to-hierarchy alignment and scale-aware guidance are well-motivated and technically sound.\n* Consistent gains in semantic metrics (CLIP 96.4%) and large inference speedup without loss of quality.\n* Deterministic generation avoids stochastic variability common in diffusion models.", "weaknesses": "* The paper claims to address the \"mismatch between fixed neural guidance ... reconstruction\", yet prior works have already explored similar ideas. Neuropictor [1] introduced ControlNet to incorporate low-level structure during decoding, and DecoFuse [2] explicitly decomposed fMRI features into \"what\" and \"where\" pathways aligned with brain hierarchies. These efforts are not discussed, and the paper frames the problem as if no prior solution existed.\n* More comparison, Neuropictor [1], MindTuner [3], and other recent approaches are missing.\n* Figure 2(b) is difficult to interpret: the design motivation and the functional role of each block are unclear, making the overall mechanism hard to understand.\n* The claimed advantage in reconstruction stability over diffusion models is not fully convincing, as many existing diffusion-based methods already mitigate stochasticity through techniques such as guidance scale tuning or ControlNet conditioning.\n* The emphasis on faster inference time is somewhat questionable. Efficiency is valuable, but its practical impact in fMRI decoding remains unclear.\n* The performance improvement over existing methods is modest, raising doubts about whether the proposed complexity yields a meaningful advance.\n\n[1] Huo et al. Neuropictor: Refining fmri-to-image reconstruction via multi-individual pretraining and multi-level modulation.\n\n[2] Li et al. DecoFuse: Decomposing and Fusing the\" What\",\" Where\", and\" How\" for Brain-Inspired fMRI-to-Video Decoding.\n\n[3] Gong et al. Mindtuner: Cross-subject visual decoding with visual fingerprint and semantic correction.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)