MoRE-Brain: Routed Mixture of Experts for Interpretable and Generalizable Cross-Subject fMRI Visual Decoding
Authors: YUXIANG WEI
Yanteng Zhang
Xi Xiao
Tianyang Wang
Xiao Wang
Vince Calhoun
Venue: NeurIPS 2025 poster ()
Keywords: brain decoding; visual reconstruction; fMRI
Primary area: neuroscience_and_cognitive_science

Abstract: Decoding visual experiences from fMRI offers a powerful avenue to understand human perception and develop advanced brain-computer interfaces. However, current progress often prioritizes maximizing reconstruction fidelity while overlooking interpretability, an essential aspect for deriving neuroscientific insight. To address this gap, we propose MoRE-Brain, a neuro-inspired framework designed for high-fidelity, adaptable, and interpretable visual reconstruction. MoRE-Brain uniquely employs a hierarchical Mixture-of-Experts architecture where distinct experts process fMRI signals from functionally related voxel groups, mimicking specialized brain networks. The experts are first trained to encode fMRI into the frozen CLIP space. A finetuned diffusion model then synthesizes images, guided by expert outputs through a novel dual-stage routing mechanism that dynamically weighs expert contributions across the diffusion process. MoRE-Brain offers three main advancements: First, it introduces a novel Mixture-of-Experts architecture grounded in brain network principles for neuro-decoding. Second, it achieves efficient cross-subject generalization by sharing core expert networks while adapting only subject-specific routers. Third, it provides enhanced mechanistic insight, as the explicit routing reveals precisely how different modeled brain regions shape the semantic and spatial attributes of the reconstructed image. Extensive experiments validate MoRE-Brain’s high reconstruction fidelity, with bottleneck analyses further demonstrating its effective utilization of fMRI signals, distinguishing genuine neural decoding from over-reliance on generative priors. Consequently, MoRE-Brain marks a substantial advance towards more generalizable and interpretable fMRI-based visual decoding.

MoRE-Brain: Routed Mixture of Experts for Interpretable and Generalizable Cross-Subject fMRI Visual Decoding
Yuxiang Wei
Yanteng Zhang
TReNDS Center (GSU, Georgia Tech, Emory) weiyuxzaaggggaeechccdi
TReNDS Center (GSU, Georgia Tech, Emory)
yntn32@outlook.com com
Xi Xiao
Tianyang Wang University of Alabama at Birmingham
University of Alabama at Birmingham xxiao@uab... edu
toseattle@siu.cm edu
Xiao Wang
Vince D. Calhoun
Oak Ridge National Laboratory wangx28ornl.gor
TReNDS Center (GSU, Georgia Tech, Emory)
vcalhoun@gatech.com edu
Abstract
Decoding visual experiences from fMRI offers a powerful avenue to understand human pereeption and develop advanced brain-computer interfaces. However, current progress often prioritizes maximizing reconstruction fidelity while over- looking interpretability, an essential aspect for deriving neuroscientific insight. To address this gap, propose MoRE-Brain, neuro-inspired framework designed for high-fidelity, adaptable, and interpretable visual reconstruction. MoRE- MoRE-Brain uniquely employs hierarchical cekture-offExperre architecture where distinct experts process fMRI signals from functionally related voxel groups, mimicking specialized brain networks. The experts first trained to encode fMRI into the frozen CLIP space. A finetuned diffusion model then synthesizes images, guided by expert outputs through novel dual-stage routing mechanism that dy- namically weighs expert contributions across the diffusion process. MoRE-BB Brain offers three main advancements: First, introduces novel Mixture-of-Expertt architecture grounded in brain network principles for neuro-decoding. Second, it achieves efficient cross-subject generalization by sharing core expert networks while adapting only subject-specifie routers. Third, it provides enhanced mech- anistic insight, as the explicit routing reveals precisely how different modeled brain regions shape the semantic and spatial attributes of the reconstructed image. Extensive experiments validate MoRE-Brans Brain's high reconstruction fidelity, with bottleneck analyses further demonstrating its effective utilization of fMRI signals, distinguishing genuine neural decoding from over-reliance on generative priors. Consequently, MoRE-Brain marks substantial advance towards more generaliz- able and interpretable (MRI-based visual decoding. Codes are publicly available:
This manuscript has been authored by UT-Battelee, L.I., under contract DF-ACDS-000R2272 with the US Department of Energy (DOE). The US government retains and the publisher, by accepting the article for publication, acknowledges that the US government retains nonexclusive, paid-up. irrevocable, worldwide license to publish reproduce the published form of this manuscript, allow others do for US government purposes. DOE will provide public access to these results federally sponsored research in accordance with DOE Public Access Plan (https: //wwu. energy gov//oe-- pitteepublic-- access access-plan).
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Introduction
Understanding brain visual systems and human perception has been a central interest in the field of neuroscience [ 2 3. The human visual system transforms perceived scenes into rich, meaningful
perception via a complex cascade of hierarchical and specialized processing [4] With the advance-
ment in the diffusion models [5] and representational space like CLIP (6] or visual prompt tuning 77],
numerous studies propose to decode visual stimuli from non-invasive brain signals such as functional MRI (IMRI) and EEG, aiming to unravel the complex mechanism under perception and advance the practical brain-computer interface. These methods usually leverage strong priors from existing generative models and train dedicated decoders to decode brain signals to frozen representational space, reconstructing natural scenes with impressive structural accuracy [8
9 10
13]
Despite this progress, significant gap persists between technical feats and neuroscientific insight. Current leading methods often treat fMRI signals as relatively monolithic inputs, lacking architec- tural designs that reflect established principles of neural processing. For instance, several works manually segment the brain into broad regions like the higher visual and lower visual cortex, then decode semantic and low-level spatial information from them separately [34 [6 17.. However, this simplification overlooks the intricate nature of the visual pathway, which involves multiple hierarchical processing stages across numerous specialized cortical areas, each handling different facets of visual information [1]] 19. Furthermore, leading research commonly encounters two critical bottlenecks: cross-subject generalization and limited mechanistic interpretability. Due to inter-individe inevvindiiilal brain variability, models often require subject-specific training. While approaches like mapping subject data shared space [ 20 21] or utilizing subject-speciffe tokens [T] exist, they typically necessitate extra training and are difficult to generalize to new subjects. More fundamentally, the predominant focus on maximizing decoding performance often obscures how different neural computations contribute to the final reconstruction. This lack of transparency hinders our ability to validate these models against neuroscientific knowledge or use them to deepen our understanding of the visual system itself.
To bridge the gap, we introduce MoRE-Brain Mixture-of-Expert Routed Brain decoder), a novel visual decoding framework that is explicitly designed to mirror the hierarchical and specialized nature of the human visual pathway [22] to overcome key limitations generalization and interpretabil- ity inherent in less biologically constrained approaches. MoRE-Brain builds upon a hierarchical cexture-of-Expeeee
(MoE) architecture first map fMRI activity patterns image and text priors. Instead of relying on single, monolithic network, BrRe-Brain employs multiple layers of "expert" subnetworks. Within each layer, trainable "router" networks learn to assign specific brain regions (voxel groups) to the experts best suited to process their signals. This allows each expert to specialize in decoding information from distinet neural populations, analogous to functional specialization in the cortex. Addressing the challenge of cross-subject generalization, we draw from findings suggesting that significant inter-individual variation arises from spatial differences in functional network topography rather than fundamental computational differences 23 24 Based on this, we hypothesize that the core visual decoding computations (the experts) can be shared across subjects, while subject-specific adjustments are primarily needed for the routers that map individual brain topographies to these experts. Our experiments validate this, demonstrating that by freezing expert weights and fine-tuning only the routers, MoRE-Bres BeR-Briin achieves strong cross-subject performance with minimal subject-specific data, offering a more scalable approach than retraining entire models.
Furthermore, effectively utilizing the disentangled information captured by these specialized experts within the iterative image generation process of diffusion models presents another challenge. This mechanism operates dynamically across both time and space: (1) A Time Router selects which hicrarchical level of expert embeddings to prioritize at different diffusion timesteps. This potentially reflects the coarsc-to-fine dynamics of visual processing, where global scene layout might be estab- lished before finer details emerge
[25]. (2) A Space Router then dynamically modulates the spatial influence of the outputs from individual experts within the selected level. This mimics how the brain might integrate features processed by different specialized areas (e.g., form, color, motion) into a spatially coherent percept [26]. Critically, the dual-routing system not only guides image generation but also provides a mechanistic lens into the reconstructing process. By observing which experts and levels are routed different times and for images with different semantic concepts, we gain insight into how distinct, modeled neural sources dynamically contribute to the final reconstruction.
Our main contributions are summarized as follows:

---

We propose MoRE-Bras Brain, an fMRI decoding framework featuring hierarchical MoE architecture explicitly inspired by the principles of functional specialization and hierarchy in the visual brain.
MoRE-Braie addresses cross- generalization by leveraging the MoE structure, sharing expert networks while adapting subject-specific routers, enabling efficient tuning for new individuals.
We introduce a novel dynamic Time and Space routing mechanism effectively integrate information from specialized experts as conditional prompts for diffusion models, reflecting potential temporal and spatial integration processes in vision.
This routing mechanism enhances interpretability, offering insights into how information from different modeled brain networks dynamically shapes the semantics of reconstructed images throughout the generative process.
2 Methodology
2.1 Preliminaries: Latent Diffusion Models and Conditioning
Our reconstruction module leverages latent diffusion models (LDMs), specifically Stable Diffusion XL (SDXL) [5]. which are generative models operating in a compressed latent space [27 28 LDMs learn to reverse diffusion process that gradually adds Gaussian noise to latent representations (zp) over discrete timesteps & 1= 1,..., T. The forward process is defined as where G₂ is noise schedule and (0, I). The core of the LDM is denoising network ca (typically a U-Net (29]), trained to predict the noise c added at timestep f, given the noisy latent 2/ and conditioning information c:
v0,20 + v1 aØe,
LLDM = Ezo.ecct [lle - c0(z,, [lle-00zz,
L, c)]1
(I)
Conditioning information e is typically derived from various modalities. For text-to-image synthesis, often consists of embeddings Tteat (Crent) from a pre-trained text encoder (e.g.. CL.IP (6). Image conditioning can be achieved using IP-Adtpeer [30], where features Tamy (Cmg) are extracted from a reference image via a pre-trained image encoder (e.g., CLIP image encoder) and mapped through an adapter network. Both Ttezt Trezi(Geert (Ctent) and Timg (Cimg) are integrated into eo usually via cross- attention mechanisms. In MoRE-Brain, we adapt this conditioning mechanism use fMRI-derived embeddings as image and text priors.
2.2 MoRF-Brain Framework Overview
MoRF-Brain employs two-stage process, illustrated Figure designed decode visual information from fMRI while incor- porating principles neural specialization and hierarchy.
Bools Me1
Hierarchical MoE for fMRI-to- Embedding Decoding: A hierarchical Mixture-of-Expert (MoE) maps input fMRI signals (J) to sets of embedding aligned with the frozen CLIP space. This stage learns to disentangle fMRI representations based on functionally specialized voxel groups.
Stage 1: Train IMRI Encoder
Stage Finetune Diffus lon Model Routers
Generated
Image
Figure 1: Overview of MoRF-Brainn: fMRI is encoded into CLIP space by the hierarchical MoE (left), guid- ing image generation via dynamic Time/Space Routers (right).
Dynamically Conditioned Image Gen- eration: We introduce novel Time and Space routing mechanism to dynamically select and integrate the multi-level expert embeddings produced in Stage I to guide the diffusion model's (SDXL's) denoising process at each timestep t. During this stage, the MoE 1MRI encoder (trained in Stage 1) frozen, while the SDXL U-Net (via LoRA (31) and the Time and Space routers are fine-tuned.
3

---

Leae-
O
Voxel Set 1 (e.qu ventral)
É Expert OH
Level
Expert
4
Router
Gronsderttc titess
(HRI
Sellct
Time Router
IHRI Router
Time Emnboding
leg. dorsal)
Expert
OP
RMR Router
Expart
Modulatt
Cress Attantion
G ranularity Level 1
Gramularity Level 2
Router
Letents
(a) fMRI MoE Encoder
(b) Time Space Router
Routing mechanism of JoRE-Brain. (a) Hierarchical fMRI MoE encoder processes voxels using routed, specialized experts across levels. (b) Time and space router that adaptively selects and modulates expert outputs.
Figure 2:
2.3 Hierarchical MoE for fMRI Encoding
To model functional specialization and hierarchy akin to the brain's visual system, we design hierarchical MoE architecture (Figure. 2h) to transform fMRI signals into sets of CLII--liggee embeddings. Given the input fMRI voxels F E RR", where is the number of voxels, instead of pre-defining brain parcels, we learn learn data-driven voxel assignments. As in Figure. a for each level of the hierarchy, a router network determines the affinity of each voxel for each expert at that level. Concretely, at level I, given input features (() (where X = F). the router computes voxel-expert affinity scores (1) uxe using learnable weight matrix w
A
W x
where is the number of experts at level I. We then apply softmax over the expert dimension to get probabilities p(') = softmax (A)).. To assign voxels to experts, we employ Top-K selection for each expert. For the j-th expert at level I, we select the indices 149 of the & voxels with the highest probability scores in the j-th column of p();
( r/") = Took(P!!) .b)
where l = - x cy. is the number of voxels assigned per expert, controlled by a capacity factor cj- We set Cf = to enable non-overlapping selection over all voxels. Each expert E.") is implemented as simple MLP, processing only the features corresponding to its assigned voxels 150)
For progressively finer specialization, outputs 0 from experts at level serve as input (I+1) to the routers at the next level (1 + 1). We start with cp = 2 experts at the first level, and each expert's output path is routed to 2 new experts the subsequent level. We use total of L 4 levels, resulting in 2 = 16 experts at the final level. This number was empirically chosen to approximate the number of distinct functional ROIs identified in recent visual cortex atlases (32] The expert outputs subsequently aligned with frozen CLIP embeddings, more details in Appendix A
2.4 Dynamic Conditioning via Time and Space Routing
Generating images via diffusion is iterative process. We leverage the hierarchical and specialized nature the MoE embeddings dynamically, which allows the model to potentially focus on different levels of abstraction (semantic vs. detail) processed by different specialized networks at different stages of generation, mirroring coarse-to-fine to-fie processing theories [33]. We introduce a time and space routing mechanism (Figure integrate MoE expert embeddings (E) into SDXL U-Net (eo).
The time router R, determines the relevance of expert embeddings from different hierarchical levels based on the current diffusion timestep t. We define learnable embeddings & [Ø1,.., Φ1]

---

representing each of the L MoE levels, and compute the relevance of each level / for each timestep t via an attention-lis mechanism:
(Wole) (Wxo)*
Pr = softmax
(4)
vds
where We and WK are learnable matrices, te is the continuous representation of t, and Pr E R contains weights indicating the importance of each level timestep t. To explicitly encourage coarse-to-fine processing, the Pr regularized by a guiding distribution Pr via KL divergence. For total 7 steps, we define Pr using Gaussian centered at μ4:
(1-m4) 20
exp
Pr.
where Me L
EV 2k=1
(k-p))2 2a
exp
Here, = controls the sharpness of the focus on the target level Based on Pr, we explore two selection methods: (1) soft selection by applying Pr to weigh the expert embedding O!!) the corresponding level and (2) hard selection by choosing level I" with the highest weight (I" = Pr..) We further provide a fixed schedule by predefining timestep intervals according each level, detailed in Appendix A.4
After selecting the relevant expert embeddings (Eael based on Ry's output), the space router R. modulates their influence spatially based on the current state of the noisy latents 2,. This mimics integrating specialized features (e.g., shape, texture) into a coherent spatial layout. We use cross- attention mechanism where 2; acts the query and the selected expert embeddings Esel act keys and values:
C softmax Won: Wx Wxesed, Wv Eset
(6)
Vdz
The resulting conditioning serves as the dynamic, fMRI-derived condition for the SDXL U-Net's cross-attention layers at timestep
3 Experiments
3.1 Settings and Implementations
We evaluate MoRE-Brain using the Natural Scenes Dataset (NSD) 34]. large-scale fMRI dataset comprising brain responses from 8 subjects viewing over 30,000 distinct natural images. We adhere
Seas Images
STOP
MindEyeV2
SPAR
Nindt ridge
MoRt-Brrin
STOP
junnetred]
MoRt-Sruin
STOP
Figure 3: Reconstructions from different methods the subject 1. More-Brain can reconstruct visual stimuli faithfully compared to baselines. An optional refinement step (see Appendix A.1.3 is employed enhance the image.

---

DreamSim
SSIM
durder
324
34 a Larsbnd
InceptionV3
CLIP Cosine Similarity
leve
ondeect
20
125 334
Figure 4: Quantitative performance across varying bottleneck sizes. MoRE-Brain shows a consistent and notable performance drop across most metrics as the bottleneck size decreases. Moreover, its performance witnesses the most significant drop after adding a bottleneck, indicating that the bottleneck is particularly detrimental to the learned rich information and the model heavily relies on the fMRI data. However, MindEye2 and MindBridge exhibit less degradation, particularly MindBridge on SSIM and Inception V3, suggesting greater influence of learned priors. Notably, MindEye2 maintains high CLIPP-Css even at extreme bottlenecks, while others show expected decline with information restriction.
to the standardized training and testing splits proposed in prior work [.. Further details on NSD and fMRI data preprocessing pipeline available in Appendix B
We compare MoREEBrrain with two methods: the recent state-of-the-art MindEye2 g and Mind- Bridge 35 To assess the extent to which models leverage fMRI information versus relying on generative priors, we adopt the bottleneck analysis proposed by [T0. evaluating performance across varying bottleneck sizes imposed on the fMRRI-deriee features. We include high-level metrics like DreamSim (36) and Inception V3 37), and low-level metric SSIM. We also measure the cosine similarity between decoded embeddings and the ground truth CLIP embedding as in [10), thereby isolating the decoder's performance from the generative model. For each method, we report a "reconstruction ceiling" (using ground truth image latents instead of fMRI-predieted latents) and "random baseline" (using random fMRI as input). More details on implementation in Appendix A
3.2
IMRI-to-Image Reconstruction
First, we evaluate the overall capabilities of fMRI-to- -image reconstruction. Figure qualitative examples of reconstructions for Subject 1, where MoRE-Brie -rain faithfully recovers visual stimuli. Quantitatively, Figure A shows average performance across all subjects. MoRE-Brain's 's performance is competitive with state-of-the-art methods. We also show reconstructions from single level of expert embeddings, see Appendix
presents
The bottleneck analysis in Figure 4 critical insights. As highlighted in the caption, MoRE- Brain's 's reconstruction quality (SSIM, DreamSim, Inception is notably more sensitive to the amount of information allowed through the bottleneck compared to MindEye2 and MindBridge. This suggests MoRE- Brain effectively utilizes the richness of the fMRI signal. On the other hand, the baselines show relative robustness narrow bottlenecks in terms of pixel-space metries, especially MindBridge. Surprisingly. MindEye2 exhibits a remarkably high random baseline CLIP-Cor Cos of approximately 0.8. and crucially, its CLIP-Cos performance remains largely consistent and high across all tested bottleneck sizes, showing minimal degradation even with severe information restriction. We hypothesize this due to MindEye2 learns strong biases towards semantic plausibility, as defined by CLIP, and overshadows the contribution of fMRI signals. Although it generates embeddings that

---

Table 2: Ablation study of the text/image conditioning and the time and space routers based on unrefined MoRE-Brain. Text conditioning only. I: Image conditioning only. (TS): Both Time and Space routers. (S): Only Space router. (T, sched): Time router with fixed schedule. (T, hard):
Time router with learned hard selection. (T, soft): Time router with learned soft selection.
T
7 T(TS)
I
ΣTTS)
T7I
(S)
TT. schod)
7IZ hard)
TT, soft)
777 (TS)
SSIM
0.410
0.402
0.422
0.382
0.407
0.375
0.403
0.397
0.402
0.415
Alex C2
0.748
0.754
0.742
0.802
0.760
0.762
0.765
0.650
0.764
0.792
Incep
0.926
0.988
0.893
0.957
0.890
0,943
0.940
0.867
0.941
0.962
DreamSSu.
0.360
0.42
0.605
0.528
0.585
0.533
0.626
0.343
0.507
make semantic sense to CLIP for any input, the structural information that depends on the fMRI is removed, leading to the relatively low random performance as measured by other metrics.
3.3 Cross-Subject Generalization with Limited Data
MoRF-Brain's separation of shared experts from subject-specific routers allows for efficient gen- eralization with limited data. After training the model subject we freeze the model's 's experts and only finetune the routers to generalize to other subjects. This massively reduces the parameters during finetuning. We compare the total parameters of MoRE-Brain and the two baselines, in Table Note that for MindBridge, we applied its 'reset-tuning" strategy.
To further demonstrate MoRE-Brain's general- izing performance with limited data, we first train the full model from scratch subject 1,
Table I:
Compare the total number of parameters and trainable parameters during finetuning.
then freeze the weights of all experts and fine-
Total Params Trainable
tune only the fMRI routers (Eq. 2 with 2.5% (1 session), 10% (4 sessions), 25% (10 sessions), 50% (20 sessions), and all new subjects' data (40 sessions). See results in Appendix
MindEve2 MindBridge
729.3M
100%
552.9M
98.48%
MoRI:-Brain
293.4M
44.84%
3.4 Ablations
To understand the contributions of MoRE- Brain's key components, we conduct ablation studies and the results are shown in Table 2 Since the cosine similarity mentioned before does not apply here, we include another low- level metric Alex (2).
Brain Volume Map Overlay 19N122
Comparing text-only (T) versus image-only (I) conditioning, we find that image conditioning generally yields superior SSIM, reflecting better structural detail. For the time router, the fixed schedule (see Appendix (A.4) enhances the re- construction compared to no router, while being worse than the learned soft selection. Overall, the results affirm the importance of both the dual conditioning streams and the dynamic routing mechanisms.
Top Ragiono on Schoofer Adaa
Figure 5: Interpretability of overall model contri- butions via ICA. Brain regions consistently con- tributing to reconstructions across 8 subjects are identified as Independent Components (ICs). ICs highlight engagement of known visual processing areas (e.g., visual central/periphereal) and higher- order association areas (e.g.. dorsal attention net- work). This demonstrates the MoRE- Bart-BBrann learns neurophysiologically plausible mappings. See Ap- pendix for complete visualizations of all ICs and ROI labels.
3.5 Interpretability: Linking Model to Brain Function
A core motivation for MoRE-Brain is to enhance neuroscientific insight. We investigate model in- terpretability at two levels: overall brain contri- butions and expert-specific functional specializa- tion. We show MoRE-Brain learns neurophysio- logically plausible mappings 3 7] with its experts learn hierarchical spatial specialization (Figure 6) and increasingly specific semantic preferences (Figure 8) mimicking aspects of brain
7

---

Level 0
Level 1
Level 2
Level 3
Epp Epi Ep3 Ep3
Epp tpl tpstes tppt Mbp
Figure 6: Visualization of voxel selectivity for individual experts within MoRE-Brain BarRnB' hierarchical MoE fMRI encoder (Subject 1, attributions projected to fsaverage surface). Each pancl shows the contributing voxels for specific expert E, 1) hierarchical specialization: Experts early levels (e-g., level 0, E0 E tend draw from broader visual regions. Experts higher levels (e.g.. level 2, level 3) exhibit finer-grained spatial specialization, focusing on more distinct sub-regions of the visual cortex and associated areas. 2) intra-level disserentiation: Within single level, different
experts (e.g., 2 vs. E(²) show preferences for different voxel populations, supporting the MoE's role in disentangling neural signals. Voxel assignments appear relatively balanced. See Appendix for detailed volume-based visualizations and ROI labels.
organization. More analyses on the routing mechanism and expert's functional specialization can be found in Appendix F-I.
3.5.1 Overall Brain Contributions to Reconstruction
To identify brain regions crucial for reconstruc- tion across subjects, compute voxel attri- butions using GradientSHAP 38 for the en- tire MoRF-Brain. The attribution maps (1000 per subject, 8000 total) are then decomposed using Independent Component Analysis (ICA) [9] into 32 spatial ICs, representing distinct co-varying patterns of functional activity con- tributing to the decoding.
Furthermore, by correlating IC activations (de- rived from the ICA mixing matrix) with the se- mantic content of the input stimuli (12 COCO supercategories [40)), we find that specific ICs exhibit significant preferences for object cate- gories (Figure 7).. For example, ICs strongly correlated with "appliance" consistently involve parieto-occipital regions. (Details on IC labeling with Schaefer atlas 41 in Appendix
Figure Semantic correlates of ICs. Heatmap shows significant (FDR corrected t-test) correla- tions between IC activations (from ICA mixing matrix) and the 12 COCO supercategories in stim- uli. Different ICs show preferential responses to semantic categories (e.g., ICs 2, 10, 18, 31 corre- lating with "appliance" map to visual and parieto- occipital areas). This links distributed brain activ- ity patterns captured by ICs to high-level semantic features. More in Appendix
G
3.5.2 Expert-Specific Functional Specialization
MoRF-Brrains hierarchical MoF architecture is designed to allow experts specialize. We

---

investigate this by examining the voxel selections and semantic correlations of individual experts. that experts progress from integrating broader visual regions at early MoE levels to focusing on more distinct voxel populations at higher levels, with clear differentiation also observed between experts within the same level.
This functional specialization extends to semantic processing. Figure g shows the correlation between expert contributions and the 12 COCO supercategories. While early-level experts show diffuse category preferences, higher-level experts increasingly specialize towards particular semantic categories. For instance, expert F (level 2) shows notable preference for "outdoor" scenes, primarily processing signals from the dorsal attention network and parts of the default mode network. Some experts remain more "universal." " These findings support the hypothesis that MoRE-Brain learns functional specialization and hierarchical processing analogous to the visual brain. (Further visualizations and analyses are provided in Appendix D.
Level Fezert Contr Burton per Cateoorry
Level 3 Expart Contribution per Cabegory
Print
E41
CH
35
Lovel L- Expat Car brosken pet Cologgry
CH
SW
S44
CH
XX
XG
155
GUV
CS
GIR
XOI
Level3
EicHtt Comt button per Calrenry
DONE
G11
20
Dret
12
C55
C15
O17
LG
2B
AZT
CH
CD
A/4
UCE
LH
2G
421
Calegory
Coteggry
Figure 8: Semantic selectivity of individual experts across MoE levels. Heatmap shows the average attribution (contribution score) of each expert F.(1) to reconstructions of images containing the 12 COCO supercategories. 1) emergent specialization: Experts at the initial level (level 0) show broad, less differentiated responses across categories. 2) higher-level selectivity: Experts at higher levels (e.g., level 2, level 3) develop stronger preferences for specific semantic categories (e.g.. EE() for "outdoor," " certain level 3 experts for "appliance" or "food"). Some experts (e.g., E some level experts) remain more "universal," " contributing to multiple categories, possibly encoding foundational visual features. This suggests an emergent functional hierarchy within the MoF.
4 Conclusion
Decoding visual experiences from fMRI remains challenged by cross-subject variability and the "black box" nature of many models. In this work, we presented MoRE-Brain, framework that directly tackles these issues through neuro-inspired design. MoRIi-Brain's 's hierarchical MoE fMRI encoder explicitly models the brain's specialized processing networks. This architecture, combined with dynamic dual-stage routing system guiding a powerful diffusion model, achieves tigh-fidelity reconstruction. More importantly, the MoE structure serves a dual purpose: it facilitates efficient cross-subject generalization by isolating subject-specins routing from shared expert computations, and it unlocks unprecedented mechanistic interpretability. Our bottleneck analyses validate MoRE- Brain's 's reliance on genuine neural information, while detailed interpretability studies reveal emergent functional specialization within the experts and highlight the dynamic contributions of different mod- eled brain regions to semantic and spatial aspects of the reconstruction. By bridging the gap between decoding performance and neuroscientific understanding, MoRE-Brain represents significant step towards developing fMRI decoding systems that are both highly adaptable and interpretable, paving the way for their use as effective tools in cognitive neuroscience.

---

References
[1] Yuxiang Wei, Anees Abrol, James Lah, Allan Levey, and Vince D Calhoun. From symptomatic pre-symptomatic: Adaptive knowledge distillation for early alzheimer's 's detection using functional mri. IEEE Transactions on Biomedical Engineering, 2025.
[2] Yuxiang Wei, Anees Abrol, and Vince D Calhoun. Hicrarchical spatio tempooall state-space modeling for fmri analysis. In Internationaa Conference on Research in Computational Molecular Biology, pages 86-98. Springer, 2025.
[3] Xi Xiao, Yunbei Zhang, Thanh-Huy Nguyen, Ba-Thinh Lam, Janet Wang, Lin Zhao. Jihun Hamm, Tianyang Wang. Xingjian Li, Xiao Wang, et al. Describe anything in medical images. arXiv preprint arXiv:2505, 05804, 2025.
[4] Giorgio Ganis, William L Thompson, and Stephen M Kosslyn. Brain areas underlying visual mental imagery and visual perception: an fmri study. Cognitive brain research, 20(2):26-241, 2004.
[5] Dustin Podell, Zion English, Kyle Lacey, Andreas Blattmann, Tim Dockhorn, Jonas Müller, Joe Penna, and Robin Rombach, Sdxl: Improving latent diffusion models for high-resolution image synthesis. arXiv preprint arXiv:2307.01990 2023.
[6] Alec Radford, Jong Wook Kim, Chris Hallacy. Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Míshkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748-8763. PmL.R, 2021.
[7] Xi Xiao, Yunbei Zhang. Xingjian Li, Tianyang Wang, Xiao Wang, Yuxiang Wei, Jihun Hamm, and Min Xu. Visual instance-aware prompt tuning. arXiv preprint arXiv:2507.07799 2025.
[8] Paul S Scotti, Mihir Tripathy, Cesar Kadir Torrico Villanueva, Reese Kneeland, Tong Chen, Ashutosh Narang, Charan Santhirasegaran, Jonathan Xu. Thomas Naselaris, Kenneth A Norman, et al. Mindeye2: Shared-subject models enable fmrii-o-image to-image with hour of data. arXiv preprint arXiv:2403. 11207, 2024.
[9] Andrew Luo, Maggie Henderson, Leila Wehbe, and Michael Tarr. Brain diffusion for visual exploration: Cortical discovery using large scale generative models. Advances in Neural Information Processing Systems, 36:75740-75781, 2023.
[10] David Mayo, Christopher Wang, Asa Harbin, Abdulrahman Alabdulkareem, Albert Shaw. Boris Katz, and Andrei Barbu. Brainbits: How much of the brain are generative reconstruction methods using? Advances in Neural Information Processing Systems, 37:54396 -54400, 2024.
[11] Qiongyi Zhou, Changde Du, Shengpei Wang, and Huiguang He. Clip-mused: Clip-guided multi-subject visual neural information semantic decoding. arXiv preprint 2rXiv:2402.08994 2024.
[12] Yizhuo Lu, Changde Du, Qiongyi Zhou, Dianpeng Wang, and Huiguang He. Minddiffuser: Controlled image reconstruction from human brain activity with semantic and structural diffusion. In Proceedings of the 31st ACM International Conference on Multimedia, pages 5899-5908, 2023.
[13] Xingjian Diao, Ming Cheng, Wayner Barrios, and SouYoung Jin. F(2If First-person statement text- to-talking face generation. In 2025 IEEF/CVF Winter Conference on Applications of Computer Vision (WACV), pages 4821 18811-8330. IEEE, 2025.
[14] Tao Fang, Yu Qi, and Gang Pan. Reconstructing perceptive images from brain activity by shape-semantic gan. Advances in Neural Information Processing Systems, 33:13038-13048, 2020.
[15] Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14453-1444 14463, 2023.
[16] Jaehoon Joo, Taejin Jeong, and Seongjae Hwang. Brain-streams. Emri-to-image reconstruction with multi-modal guidance. arXiv preprint arXiv:2409. 0204409..2999 /2099, 2024.
[17] Xingjian Diao, Chunhui Zhang, Weiyi Wu, Zhongyu Ouyang. Peijun Qing. Ming Cheng, Soroush Vosoughi, and Jiang Gui. Temporal working memory: query-guided segment refinement for enhanced multimodal understanding. arXiv preprint arXiv:2502.0600 2025.
[18] Carolyn Jeane Perry and Mazyar Fallah. Feature integration and object representations along the dorsal stream visual hierarchy. Frontiers in computational newroscience, 8:84, 2014.
10

---

[19] Alyssa A Brewer, William A Press, Nikos K Logothetis. and Brian A Wandell. Visual areas in macaque cortex measured using functional magnetic resonance imaging, Journal Neuroxcience, 22(23): 10416- 10426, 2002.
[20] Matteo Ferrante, Tommaso Boccato, Furkan Ozcelik, Rufin VanRullen, and Nicola Toschi. Through their eyes: multi-subject brain decoding with simple alignment techniques. Imaging Neuroscience, 2:1 2:1-21, 2024.
[21] Paul Scotti, Atmadeep Banerjee, Jimmic Goode, Stepan Shabalin, Alex Nguyen, Aidan Dempster, Nathalic Verlinde, Elad Yundler, David Weisberg, Kenneth Norman, et al. Reconstructing the mind's eye: fimri-to- image with contrastive learning and diffusion priors. Advances in Neural Information Processing Systems, 36:24705-24728, 2023.
[22] Daniel J Felleman and David C Van Essen. Distributed hicrarchical processing in the primate cerebral cortex. Cerebral cortex (New York, NY: 1991), 1(1):1-47, 1991.
[23] Kevin M Anderson, Tian Ge, Ru Kong, Lauren M Patrick, R Nathan Spreng, Mert R Sabuneu, BT Thomas Yeo, and Avram Holmes. Heritability of individualized cortical network topography. Proceedings of the National Academy of Sciences, (18()):2001227110 11811 2021.
[24] Hannah S Savage, Peter CR Mulders, Philip FP Van Eijndhoven, Jasper Van Oort, Indira Tendolkar, Janna N Vrijsen, Christian F Beckmann, and Andre F Marquand. Dissecting task-based fmri activity using normative modelling: an application to the emotional face matching task. Communications Biology, 7(1):888, 2024.
[25] Rolf Skyberg, Seiji Tanabe, Hui Chen, and Jianhua Cang. Coarse-to-fine processing drives the efficient coding of natural scenes in mouse visual cortex. Cell reports, 38(13), 2022.
[26] Melvyn A Goodale and A David Milner. Separate visual pathways for perception and action. Trends in neurosciences, 15(1):20-2) 25, 1992.
[27] Jonathan Ho, Ajay Jain, and Pieter Abbecl. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840 6851, 2020.
[28] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High-resolution image synthesis with latent diffusion models. In Proceedings of the IEEF/CVF conference on computer vision and pattern recognition, pages 10684-10695, 2022.
[29] Olaf Ronneberger, Philipp Fischer, and Thomas Brox. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assistedr intervention -MICAI 2015; 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, pages 234-241. Springer, 2015.
[30] Hu Ye, Jun Zhang, Sibo Liu, Xiao Han, and Wei Yang. Ip-adapter: Text compatible image prompt adapter for text-to-image diffusion models. arXiv preprint arXiv:2308.06721, 2023.
[31] Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation large language models. ICLR, 1(2)3, 2022.
[32] Mona Rosenke, Rick Van Hoof, Job Van Den Hurk, Kalanit Grill-Spector, and Rainer Goebel. probabilistic functional atlas of human occipito-temporal visual cortex. Cerebral Cortex, 31(1):603- 619, 2021.
[33] Valerie Goffaux, Judith Peters, Julie Haubrechts, Christine Schiltz, Bernadette Jansma, and Rainer Goebel. From coarse to fine? spatial and temporal dynamics of cortical face processing. Cerebral Cortex, (17))667- 476, 2011.
[34] Emily J Allen, Ghislain St-Yves, Yihan Wa, Jesse L Breedlove, Jacob S Prince, Logan T Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, et al. A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence. Nature neuroscience, (661):11-1126, 2022.
[35] Shizun Wang, Songhua Liu, Zhenxiong Tan, and Xinchao Wang. Mindbridge: A cross-subject brain decoding framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 11333-11342 2024.
[36] Stephanie Fu, Netanel Tamir, Shobhita Sundaram, Lucy Chai, Richard Zhang, Tali Dekel, and Phillip Isola. Dreamsim: Learning new dimensions of human visual similarity using synthetic data, arXiv preprint arXiv:2306. 09344, 2023,
11

---

[37] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens. and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings the IEEE conference on computer vision and pattern recognition, pages 2818-2826, 2016.
[38] Scott M Lundberg and Su-In Lee. A unified approach interpreting model predictions. Advances in neural information processing systems, 30, 2017.
[39] Aapo Hyvärinen and Erkki Oja. Independent component analysis: algorithms and applications. Neural networks, 13(4-5):411 14340,13,,,,,, 2000.
[40] Tsung-Yi Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. Computer vision-ECCV 2014: 13th European conference, zurich, Switzerland, September 6-12, 2014, proceedings, part 13, pages 740-755. Springer, 2014.
[41] Alexander Schaefer, Ru Kong, Evan M Gordon, Timothy O Laumann, Xi-Nian Zuo, Avram Holmes, Simon B Eickhoff, and BT Thomas Yeo. Local-gu Local-global parcellation of the human cerebral cortex from intrinsic functional connectivity mri. Cerebral cortex, 2809):3095-31 14, 2018.
[42] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarezyk, and Jenia Jitsev. LAION 5b: An open large-scale dataset for training next generation image-text models. Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[43] Sungnyun Kim, Gihun Lec, Sangmin Bac, and Se- Young Yun. Mixco: Mix-up contrastive learning for visual representation. arXiv preprint axX1v:201000330,, 2020.
[44] Yuxuan Ding, Chunna Tian, Haoxuan Ding, and Lingqiao Liu. The clip model is secretly an image-to- prompt converter. Advances in Neural Information Processing Systems, 36:56298-56300. 2023.
[45] Jacob Prince, lan Charest, Jan W Kurzawski, John A Pyles, Michael Tarr, and Kendrick N Kay. Improving the accuracy of single-trial fmri response estimates using gimsingle. Elife, 11:775999, 2022.
12

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: In this paper, we propose new framework for iMRI-based visual decoding. We discuss our main claims and contributions in the abstract and introduction.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much results can he expected to generalize to other settings.
It is fine to include aspirational goals as motivation long as is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss our limitations and future work in Appendix K
Guidelines:
The answer NA mcans that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well- speciflcciiii asymptotic approximations only holding locally). The authors should reflect how these assumptions might he violated practice what the implications would
The authors should reflect on scope the claims made, e.g., the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to text system might not be used reliably to provide closed captions
online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't It acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity the community. Reviewers will be specifically instructed not penalize honesty concerning limitations.
3. Theory assumptions and proofs
For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?
Question:
Answer: [Yes]
Justification: This paper does not propose completely new theory. However, we thoroughly discuss the rationales behind our designs, with clear equations and assumptions if available.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and croseeeeferenced.
13

---

All assumptions should be clearly stated or referenced in the statement of any theorems.
The proof' can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: provided detailed information about the implementations and settings of our method. The data are publicly available. We will release a public code repository for reproduction.
Guidelines:
The answer NA means that the paper does not include experiments.
the paper includes experiments. No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing model checkpoint, or other means that are appropriate to the research performed.
example, contribution
While NeurIPS does not require releasing code, the conference does require submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. a large language model), then there should either be a way to access this model for reproducing the results way to reproduce the model (c.g.. with an open-source dataset instructions for how construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access the model is limited in some way (e.g.. registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described supplemental material?
Answer:
[Yes]
Justification:
The dataset used in the paper is publicly available. We will release the relevant code.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines httxpe://nips.cc//pee cc/public/
Galdess/CoeeSubni httpee//oooeebbaaiiooooooooooooe
for more details.
While we encourage release of code and data, we understand that this might not possible,
so "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g.. for a new open-sourg benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: : //tp////...../////// cc/public/ out.dess/CooeeSuunleesssooooPo.. for more details.
14

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from script and why.
At submission time, preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental sefting/details
Question: Does the paper specify all the training and test details (c.g,, data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: We include details on the training of our model.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [Yes]
Justification:
We present numerical experiment results to justify our method.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should given (e-g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error than state that they not verified.
CL, if the hypothesis of Normality of errors
asymmetric distributions, authors should be careful not show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). crror bars are reported tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, docs the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Answer: [Yes]
Justification:
We present implementation details in appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
15

---

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform. every respect, with NeurIPS Code Ethics
http://wwww.p..cpbbbbiieeeeeeeee [Yes]
Answer:
Justification:
Yes, we comply with NeurlPS's Code Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e-g., if there is a special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive socictal impacts and negative socictal impacts of the work performed?
Answer: [NA]
Justification: The paper does not relate to possible socictal impats or biases.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no socictal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations. (e.g.. deploy- of technologies that could make decisions that unfairly impact specific groups), privacy considcrations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated relcase of models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible relcase of data or models that have a high risk for misuse (e.g. pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Not applicable to our method.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but encourage authors take this into account and make best faith effort.
16

---

12. Licenses for existing assets
Question: Are creators or original owners assets (e.g., code, data, models), used the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected"'
Answer: [Yes]
Justification: we cite the related works mentioned in the paper.
Guidelines:
The answer NA means that the paper does not use existing assets.
authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, possible, include URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, ppaperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
this information not available online, the authors are encouraged reach out asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We present a new method and will release the code.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model as part of their sub- missions via structured templates. This includes details about training. license, limitations.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, applicable, as as details about compensation (if any)?
Answer: [NA]
Justification: No crowdsouring included.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection. curation, other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: Not applicable.
Guidelines:
17

---

The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
initial submissions, not include any information that would break anonymity applica- ble), such as the institution conducting the review,
16. Declaration LLM usage
Question: Does the paper describe the usage of LLMs it an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is required.
Answer: [NA]
Justification: The method development does involve LLMs.
Guidelines:
The NA that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https:/ httppp///eeaiipe..ccc/eeeeeeeeee http://cccc.ooeeeeeeee/////0//10 should or should not be described.
for what
A More Details on MoRE-Brain
A.1 Training and Inference
MoRE-Brain's training is conducted in two distinct stages:
A.1.1 Stage 1: Training the Hierarchical MoE IMRI Encoder
the first stage, we train the hicrarchical Mixture-of- Experts (MoE) fMRI encoder (described in Section 2.3) to map fMRI signals to the embedding spaces of the frozen CLIP image (ViT-bigG/14) and text encoders (from
LAION-2B [2 6).. The training objective for this stage adapts losses from S and incorporates load balancing loss for the MoE routers:
Estggl CMSE (PredCL.P, GTcLi) Co (Predcuu GTaP)
03
c
(7)
1=0
where:
GTcur represents the ground truth CLIP image and text embeddings corresponding the viewed stimulus.
PredeLIP the set of predicted CLIP embeddings generated by the MoE IMRI encoder. To obtain single prediction the loss calculation this stage, we aggregate CLIP-aligned embeddings produced by the experts. Specifically, for each level the embeddings {c .g < ) from all its
e experts are averaged. Then, these level-wise average embeddings themselves averaged across levels to produce the final PredecIP:
Pred
CLIP
typo
(
where type either 'img' or 'text'. This aggregation is used for computing LASE and Leonnative
LasE is the Mean Squared Error loss.
Lcomanive is a contrastive loss (i.e.. InfoNCE) incorporating MixCo augmentation 44 and soft target labels (derived from the dot product similarity matrix of ground truth CLIP embeddings within batch), similar SofiCLIP in (8. This loss is applied separately for image and text embeddings.
18

---

Table 3: Using text embeddings projected from image embeddings to condition the diffusion model.
SSIM
Alex (2)
Incep
DreamSim
baseline
0.415
0.792
0.962
0.507
project-based text conditioning
0.384
0.773
0.941
0.542
LB load balancing loss for the router level I, encouraging a even distribution of voxel
assignments across its ej experts. It defined hased the fraction of voxels ,,()) assigned to each expert j at level l:
LB
Σ
1 61
where 1,th count of voxels routed to expert j at level total number of voxels v
(9)
We empirically d: = 1, 02 0.33, and az 0.1.
A.1.2 Stage 2: Fine-tuning SDXL with Time and Space Routers
second stage, the pre-trained MoE fMRI encoder from Stage frozen. We then train Time and Space Routers (Section 2.4) and fine-tune the SDXL diffusion model. Low-Rank Adaptation (LoRA) [3] with a
rank of 16 is applied to the cross- aussettttntionn layers of the SDXL U-Net to enable efficient tne-tuning. The training objective for this stage is the standard LDM denoising loss (as in Eq. [
Sugee cz [||c - co (2, t, C)MRR))|
(10)
where CJMRI Rs is the conditioning derived from fMRI signals F via the frozen MoE encoder and the trainable Time and Space Routers, as described Section 2.4. each training iteration, random diffusion timestep sampled.
A.1.3 Inference
During inference, the frozen MoB fMRI encoder processes the input fMRI data to produce multi-level sets of CLIP embeddings (+0)) = {(4g) img fezt
4t ) for each level E (O...., L 1}. The Time Router selects a level , and the Space Router then uses the embeddings conditioning C/MRR for the fine-tuned SDXL model at each denoising step. We classifier-free guidance
((²)) from this chosen level to generate the dynamic
scale of and maximum 30 denoising steps for image generation.
After generating the image, we include an optional refinement step as in g. We use the refinement model of SDXL, with the same text guidance as the first generation, to refine the model. We empirically find this slightly enhances the reconstruction quality (see Figure 4).
A.2 fMRI-to-CLA CLIP Text Embedding
The text captions associated with stimuli (i.e. COCO captions) vary in length. The full CLIP text embedding. typically CLIP tea E Mikken DCLIP (e.g.. Neokens 77, DELIP 1280), contains information distributed the first few token positions. However, 44 suggests that conditional diffusion models, the embedding of the sequeece (EOS) token often carries a significant portion of the overall semantic information. Based on this, for our IMRI-to-tess text decoding, we primarily focus on predicting the EOS embedding from the fMRI signal: Predgos E R DeLIP, During Stage 1 training, the Preduc (Eq. 8 is trained to match the ground truth EOS embedding of the corresponding caption.
To use the predicted EOS embeddings for conditioning, expand the single predicted EOS vector Predpos fill the first Niakens - positions (i.e.. positions) the text conditioning tensor. final token position then filled with a fixed start-of -euuueeeeenceeeee (SOS) token obtained from the frozen CLIP model. This approach empirically yields strong decoding performance while reducing the complexity of the IMRI-to-tex. mapping task.
A.3 Alternative: Projecting Image Embeddings to Text Space
Inspired by Fr which demonstrated that CLIP image embeddings can be effectively projected into the CLIP text embedding space via a simple linear transformation, we also explore an alternative strategy to generate text conditioning without training another decoder. In this setup, we directly use the closed- fosed-form linear transformation matrix from [44) to generate Predgos. Our experiments in Table Blndiiate that this projection method achieves comparable performance to direct IMRI-to-ted decoding.
19

---

A.4 Fixed Time Routing Schedule
The human visual system is understood process scenes in coarse-to coarse-to-fine manner time. Inspired by this, one the strategies for our Time Router (Section 2.4) is fixed schedule that deterministically selects an expert level 1" based on the current diffusion timestep t.
Given L hierarchical levels in the MoE IMRI encoder E {0, I,..., L-1)) and a total of diffusion timesteps (t E {0, 10......77 T 1}), the fixed routing schedule selects the level 1* as follows:
1" min (1
L. T
L (
(11)
This schedule proportionally maps the normalized current timestep (t/T) to the available MoE levels. Earlier
timesteps (closer to 0) will select lower-indexed levels (e.g., level 0), which are hypothesized to capture coarser, more global information. Later timesteps (closer T - 1) will select higher-indexed levels (e.g.. level L 1). hypothesized to capture finer details. The min(-, L- - 1) ensures the index stays within the valid range [0, 1 The fixed schedule serves as a simple baseline to validate our intuition over the coarse-to- selection strategy, as in Table 2
B NSD Dataset and Preprocessing
The fMRI signals are beta weights estimated from each session by GLMSingle 45). We use the preprocessed fMRI voxels in 1.8mm volume space, masked by the "nsdgeneral" ROI mask provided by the NSD dataset. IMRI from subjects 1, 2, 5, and 8 contains 40 sessions, while the other subjects have about 30 sessions. To ensure consistent training data volume across subjects for certain analyses and to simplify batching, we balanced the training sample counts. For subjects with fewer samples, a subset of their existing samples was randomly selected and repeated to match the count the subject with the maximum number of samples. This balancing was applied only where strictly necessary for model training architecture, while test performance was always evaluated on unique trials. The test set consists of images from NSD's "shared 1000" subset, comprising 1000 images viewed by all 8 subjects, ensuring comparability. Following established practices (8) applied z-score normalization to the fMRI voxel activities. This normalization was performed separately for the training and sets within each subject to prevent data leakage.
C MoRE-Brain's Generalization Ability
We first present the results of finetuning MoRE-Brain's fMRI routers to generalize it to the other 7 subjects, Figure We first train the model from scratch based on subject then finetune only the fMRI routers in each level of MoE. The time and space routers trained based on subject I are directly used during inference on other subjects.
SSIM
InceptionV3
0.43
1
0.55
0.39
0.37
0.38
0.31
0.7
0.5
0.25
05
smax
Issalre
Sudf7
-4-5482
S.B5 Se-Stdji
- Sugg
-e-SUk) - SugB
CLIP Cosine Similarity
DreamSim
085
0.88
0.8
0.81
0.75
0.7
088
0.55
0.70
0.76
035
0.72
0/43
3 1555
4.505
5e55
60 9e5s
baseline
3 19550
$055
s055
40 sess
bocelre
Figure 9: Generalize MoRE-Brain to other 7 subjects by finetuning only IMRI routers using different amounts of data. "baseline" denotes to retrain the model from scratch for the new subject.
20

---

Expert 0
Expert
Expert
Expert
Expert
Level Expert
Figure 10: Predicted visual saliency maps for experts MoE levels 0 and 1. Brighter regions indicate higher predicted importance to the expert. Saliency maps often focus on primary objects. Differences between experts the same level (e.g., between Fgg and Fz( for the same image) highlight their specialized visual field focus.
D Image Saliency Predicted by MoRE-Brain
To visualize which parts of the input image most influential for individual experts within MoRE-Bran we project the voxel-space attribution scores for each expert onto pixel space the stimulus image. This transformation leverages the population receptive field (pRF) parameters provided by NSD dataset. Each pRF model characterizes a voxel's response as 2D Gaussian sensitivity profile in the visual field, defined by its center coordinates (zo, (o) and size (standard deviation σ). For each voxel, its attribution value (calculated by GradientSHAP) distributed
pixel grid representing the visual field. This distribution weighted by the final
voxel's pRF Gaussian profile: the attribution maximal at pRF center and decreases with distance. saliency value for each pixel in the image is the sum of these weighted attributions from all voxels that have a pRF overlapping that pixel. This process effectively translates the expert-specific neural importance patterns from the fMRI voxel space back into human-inite interprecablle visual saliency map.
Figure TO illustrates these predicted visual saliency maps for individual experts at the first two levels of the MoE hierarchy. The resulting salieney maps often highlight regions corresponding the main object(s) or salient parts of input image. Notably, while the saliency fields different experts within same level overlap (as expected, since they are processing the same overall stimulus), they also exhibit clear divergences. These differences suggest that individual experts, even at the same hierarchical level, are learning focus on distinct aspects spatial regions of the visual input, driven by the different underlying neural populations they are routed from the visual cortex. This supports the notion of functional specialization among experts.
E Reconstructions from Single Level of Experts
To investigate the nature of information captured at different hierarchical levels of the MoE IMRI encoder. conducted an experiment where image reconstructions were generated using only expert embeddings single, specific level. For this analysis, the embeddings from all experts F.S)) within chosen level I (i.e.,
(()) {(c) img ) were averaged to form a single pair image and text conditioning vectors SDXL model. The Time and Space Routers were bypassed in this setup.
21

---

Level 0
Level
Level
Level
Figure 11: Qualitative examples of image reconstructions conditioned on expert embeddings from a single MoE level. Lower levels (coarse granularity) create reconstructions that may have wrong semantics, while higher levels reconstruct images more accurately.
F
Analysis of the Time and Space Routers
F.1
Expert Utilization
The Space Router (R.s) dynamically integrates emheddings from the experts of selected MoE. level to condition the SDXI. U-Net (Fq. [). Ideally, for given selected level I", the Space Router should effectively utilize the information from all experts that level, least show balanced pattern of utilization all experts expected contribute.
assess this, we analyzed the attention weights assigned by the Space Router to embeddings of individual experts within the selected level /". Specifically, in the cross-attention mechanism of the Space Router (where z is query and Ese! are keys/values), the attention scores over the et- experts indicate their contribution to the final conditioning CyMRI. We averaged these attention scores for each expert across all diffusion timesteps and all test samples which its level was selected by the Time Router.
Figure 12 displays the average utilization (mean attention score) for each expert within each of the MoE levels. Within most levels, expert utilization relatively balanced, suggesting that the Space Router generally draws information from all available experts at the selected level. Furthermore, some experts exhibit slightly higher average utilization (e.g., experts 2 and 3 in level 1). This could indicate that these experts either capture slightly more consistently relevant information that the router develops slight preference.
F.2 Granularity Selection by the Time Router
The Time Router (R.) is designed to select the most relevant MoE hicrarchical level (granularity of fMRI features) each diffusion timestep potentially mirroring coarse-to- fine visual processing dynamies. We analyzed the hehavior of the Time Router when configured for *soft selection where outputs attention weights Pr TRL over L levels (Equation These weights indicate the preference for each level a given timestep.
Figure 13 shows these attention scores Pr, averaged across all test images, for each MoE level as a function of the normalized diffusion timestep (t/l). The plot reveals clear temporal dynamic where the router assigns higher weights for lower levels early timesteps, and shifts its preference to higher levels at late timesteps. The Time Router learss prioritize global scene structure and semantics encoded by lower-level experts during the initial stages of image generation, and then progressively incorporates finer details from higher-level experts as the generation process refines the image.
22

---

Figure 12:
Average utilization of experts within each of the 4 MoE levels by the Space Router. Utilization measured as the mean attention score assigned an expert's embedding when its level is selected. Experts within each level generally show balanced utilization, suggesting effective integration of information from all specialized subnetworks.
Weights
(Coarsest)
0.25
mua)
Altention Weight
0.00
Router Preference Across Diffusion Process (Binned)
Level & (Courssat)
Lesel 2
Lesel
Level 3 (8 neet)
0.150
Aweroge Attention Weight
0.22
0.50
demasinng
danobenng
0375
0.00
0.2
22
24
6.5
Normaized Timastep [1=earliest, D-latest)
Figure 13: Time Router's learned preference (average attention weights Pr) for MoE granularity levels across normalized diffusion timesteps (1/T). Level 0 (coarsest) to Level (finest). The router exhibits clear coarse-to-fine selection pattern: lower levels (LO, LI) are preferred during early diffusion timesteps (left side of plot), while higher levels (L2, L3) are increasingly preferred during later timesteps (right side), guiding the generation from global structure fine details.
23

---

Brain Volume Map Overlay on MNI152
Top Regions on Schaefer 2018 Atlas
100
Jm-38
2-22
r-M
50
x-38
X020
Xe-14
xe-44
Figure 14: Comprehensive 2D slice visualizations for all 32 Independent Components (ICs) derived from SHAP attributions, corresponding Fig.
Figure 15: Top functional brain regions (Schaefer 2018 atlas, 1000 ROIs) associated with prominent Independent Components (ICs).
24

---

Table 4: Schaefer 2018 atlas label components and abbreviations
Type
Abbreviation
Explanation
Hemisphere
LH
Left Hemisphere Right Hemisphere
RH
Network Name
X""
Default Mode Network X (a sub-network of the DMN)
DorsAttn"X "X"
Dorsal Attention Network X (a sub-network of the DAN) Central Visual Network
VisCent
VisPeri
Peripheral Visual Network
Anatomical
Landmark/Region
IPL
Inferior Parietal Lobule
PHC
Parahippocampal Cortex
ParOcc
Parietal-Occipital region
TempOcc
Cemporo-Occipital region (junction of temporal and occipital
lobes)
ExStr
Extrastriate Cortex (visual cortex areas outside the primary visual cortex)
Striate
Striate Cortex (Primary Visual Cortex, VI)
StriCal
Striate and Calcarine Cortex (V1 area around the calcarine sulcus)
ExStrinf
Inferior Extrastriate Cortex
G ROIs for MoRE-Brain
We further show the complete 2D slices of Figure. (see Figure 14aad the top regions labelled by Schaefer
2018 atlas that contribute to the reconstruction (see Figure 15 Explanations of abbreviations are in Table
H Inter-Subject Variability in ICA Component Expression
The Independent Component Analysis (ICA) performed on the combined GradientSHAP attribution maps yields mixing matrix. This matrix quantifies the extent which each Independent Component (IC) is expressed in the attribution map of each specific image for each subject. To investigate inter-subject variability in how these neuro-computational patterns (represented by ICs) are utilized, we calculated average mixing coefficient for each of the 32 ICs, separately for each of the 8 NSD subjects.
Figure 16 illustrates these subject-specific average IC expressions. The heatmap reveals significant inter-subject variations. For instance, Subject shows strong average expression of IC15, IC17, and IC27, whereas Subject 3 strongly expresses IC8. These differences suggest that while the underlying computational components (ICs) identified by MoRE-Brain may he common, their degree of engagement prominence vary across individuals during visual decoding task. This aligns with known inter-individual differences brain function and anatomy and underscores the importance of subject-specific adaptation, which MoRE-Bran addresses through its router mechanism.
25

---

Average Mixing Coefficients Subject
Sulpect
Suntec
Subject's Subjec susjets
Sultecs
Sultect 5. Soltect Subject
22N
22
Componentts
Figure 16: Inter-subject variability in the average expression of 32 Independent Components (ICs). Each cell represents the mean mixing coefficient of an IC (columns) for a given subject (rows). The distinct patterns across rows highlight that subjects differ in their reliance on expression of these common functional components identified from fMRI attributions, motivating subject-specine model adaptation.
Differential Brain Activity for Paired Semantic Categories
To further explore how MoRF-Brain utilizes different hrain regions for processing distinct semantic content, we analyzed the ICs most strongly associated with specific pairs of image categories. For each category in pair, identified the top ICs based their average mixing coefficients for images belonging that category. These were then mapped to the Schaefer 2018 atlas identify the corresponding functional brain regions. selected three pairs for comparison: "electronic" vs. "vehicle", "animal" vs. "person", and "outdoor" vs. "sport", with the last pair being more semantically similar than the first two.
26

---

1.1
Electronic vs. Vehicle
Electronic
Vehicle
IC 24
IC
AL WWW.LESARRLIII WWA.REIENNOOOOOCCOCOOMOM WOOOTLLGER SIA.EEILLLLLLLL SWILIIOLL DO DWW.RCIIIGG AALLIAA WWW.WROREEEEE
MUNGPLLSRE Densav PL 1il Wecer bet. DeoieAA
IC
IC
Sonw porved sole yourd tpe Weowe yvay, MREWN PANN, TREH SINNG PANEEE
PeSEInoRe Borty Doodaaa pupty yoont your pager Dannttee PARti, SIOINE
IC
Figure 17: Comparison of top contributing Independent Components (ICs) and associated brain regions (Schaefer 2018 atlas) for "electronic" (left panels) versus "vehicle" (right panels) stimuli. Electronic objects significantly engage visual pathways (VisCent, VisPeri) for detailed analysis, attentional networks (DorsAttm), and memory-related regions like the Parahippocampal Cortex (PHC within Default Mode Network), potentially reflecting detailed feature encoding and retrieval of functional knowledge. Vehicle stimuli also heavily recruit visual and attentional networks, but are particularly distinguished by widespread and strong activation of the Dorsal Attention Network ADorsAttnA/B), suggesting greater emphasis on spatial processing, implied motion, and interaction with the environment.
27

---

1.2
Animal vs. Person
Animal
Person
Ue bode 401 Deins te APP.APP Dww.ttrd fwrtee WiML
IC
2
IC 30
yeu puthi, Me 2R bestand petu.
IC Furic
IC
Deniina
IC 27
IC 11
Figure 18: Comparison of top contributing ICs and associated brain regions for "animal" (left panels) versus "person" (right panels) stimuli. Both categories robustly activate extrastriate visual areas (e.g., VisCent_ExXtrr) Processing of person images shows more prominent involvement of Parahip- pocampal Cortex regions (DefaultC_Pr) PHC) within the Default Mode Network, potentially indicating enhanced episodic memory retrieval and social cognitive processes. Animal images also engage the Default Mode Network, notably via regions like the Inferior Parietal Lobule (DefaultC IPL), possibly related to contextual association or theory of mind precursors.
28

---

1.3
Outdoor vs. Sport
Outrloor
Sport
RL Donvenn SMOTESSP SPALL Www. LAS.A LAALL AAALINN Www.ECEECCCCCCCC MAL.ANULUSEE
IC 17
IC 21
26
IC 27
IC 14
Figure 19: Comparison of top contributing ICs and associated brain regions for "outdoor" (left panels) versus "sport" (right panels) stimuli. These semantically related categories show consider- able overlap their neural correlates. Both robustly engage: (1) Visual networks (VisCent ExStr, VisPeri_Ess Ersti_fxStnnf for detailed scene analysis; (2) The Dorsal Attention Network (DorsAttmA, DorsAt- tnB) for processing spatial layouts and actual implied actions; and (3) Regions of the Default Mode Network (e.g., DefaultC _PFL) likely involved in contextual understanding and memory associations. Subtle differences may exist in the relative weighting of these networks.
Hierarchical Functional Specialization of MoE Experts
We further provides detailed visualizations and descriptions of the functional specializations learned by the experts at level MoRE-Bran hierarchical MoE fMRI encoder. The analyses are based voxel attribution maps for each expert, mapped to the Schaefer 2018 atlas. reflect the varying granularity and potential information integration across levels, we visualize different number of top contributing ROIs for experts different levels (e.g., top ROIs for level 0, progressively fewer for higher levels, down top for level 3).
29

---

Expert
Expert
Dorsidaa, Boraitee, Farbue, 2 WaCeet Extly Airsitts,, TempOcc, [L) Macant,
P. waCart Vaccett C.dry
RIN, Dwt.uitccCCCC. PL, LM, Valher Ddtantt www.aa. Eodkt 28 Aonk PurBic,
Figure 20: Functional specializations of experts at level O. Experts at this initial level draw broadly from the visual system. Both experts show contributions from regions within the bilateral Dor- sal Attention Network and the Visual Central Network, indicating processing across the visual hierarchy. F additionally utilizes regions the bilateral Visual Perisylvian Network, suggesting possible emphasis on broader spatial orienting or integration with internally-guider attention along- side core visual processing.
Expert
Expert 1
tesient
Expert 2
Expert 3
Figure 21: Functional specializations of experts at level 1. Level 1 experts exhibit clearer divergence compared to level O. For example, E(() appears to specialize in processing signals from dorsal stream areas (spatial/motion, 'where' pathway), while E may focus more ventral stream areas (object recognition, 'what' pathway), potentially with hemispheric biases .Other experts like E(") and E(I) might integrate information from both streams under attentional control, possibly with differing
hemispheric emphases (e.g., (x( recruiting bilateral attention including left SPL, ES") showing strong right hemisphere Dorsal Attention involvement).
30

---

Experto
Expert
Traythu, PAA Beuttad, 2uttic
Denaiiadl, Aodia ZIMM Dandtted TempBau
Expert
Expart
HATerphec
Expert d
Expert
Expert
Expert?
DeSSiIn THE POCK LILLLIEEE Varnet Do0en
BR. DESERNA JILE Dew.EEA SPILL
Figure 22: Functional specializations experts at level 2. Experts appear to refine the specializations from level 1, applying more nuanced and context-dependent attentional modulation. For instance, specific experts might focus on outputs from 'what' or 'where' pathway specialists from LI and further modulate these based on internal cognitive factors This level could represent stage of integrating specialized visual information with broader cognitive contexts.
complex attentional strategies.
31

---

Expert
Expert
Terpōrc_
Expert
Expert 6
LH, Dww.aarknaaaaoom
Expert 8
Expert 10
EN Defasosss PL.
LM, Darskttan SPL
Expert 12
Expert 14
Exitry2
ParOce_2
Figure 23: Functional specializations of experts level 3. Experts demonstrate fine- roeegrained spe- cialization. The depicted experts show distinct fractionation of the Dorsal Attention Network into specific regional and hemispheric components, suggesting focus on precise aspects such as spatial orienting (e.g., RH DorsAttnA_ ParOCccaaPCC__1 2; DorsAttnA ParOcc_ 1), higher-order visual attention and integration (e.g., RH DorsAAAn T_eep_eeeeeeeeeeeeeeeeeeee __, and top-down attentional control or spatial working memory (e.g.. LH__DorAttnA__PP specialized processing units at the apex of hierarchy.
This highlights the emergence of highly
32

---

K
Limitations
While MoRE-Bratd demonstrates significant advances in interpretable and generalizable IMRI decoding, several limitations should be acknowledged.
First, approach relies fMRI BOLD signals processed using GLM-Single to estimate activity betas for each stimulus presentation. This methodology inherently averages neural activity over the duration of the hemodynamic response, thus omitting the fine-grain graiird temporal dynamics present the raw fMRI time series and potentially faster neural processes. Consequently. MoRE-Brain decodes relatively static representation of neural activity associated with a stimulus.
Second, the primary input features are derived from voxels within the 'nsdgeneral' ROL, which predominantly captures visually responsive cortex while potentially excluding contributions from other functional networks involved in higher-level cognition, memory retrieval, or contextual modulation that might also shape perception. As a result, MoRE Brain is primarily optimized to decode the visual content directly driven by the stimulus ("what we see") rather than potentially more complex or internally generated perceptual states ("what we think" imagine).
Finally, while our bottleneck analyses demonstrate that MoRF-Brain relies actual neural signals rather than generative priors, still diffusion model previous works generate images. This potentially restricts us from correlating hrain regions with spatial details the image. Training an autoregressive model from scratch and directly mapping voxels to pixels may further enhance our understanding of the brain visual system.
33


Reviews:
[{"review_id": "uZ9pl5gOQN", "reviewer": "Reviewer_nsMo", "summary": "This paper introduces MoRE-Brain for fMRI-based visual reconstruction. It employs a Mixture-of-Experts (MoE) architecture where distinct \"expert\" models are assigned to process signals from functionally related brain regions.", "questions": "See the weaknesses.", "limitations": "N/A", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- The paper proposes a Mixture-of-Experts (MoE) architecture where each expert is dedicated to processing signals from a distinct, functionally related brain region.\n- The paper is clearly written.\n\nWeaknesses:\n- The claims regarding \"interpretability\" in the title and abstract appear to be overstated. It is currently unknown whether the brain's visual pathway actually operates in a manner analogous to the dynamic routing mechanism proposed in the paper. If this neuro-inspired claim is based on existing literature, the authors should provide the relevant citations to support it.\n- The experimental evaluation is missing a quantitative comparison with prior work on several standard metrics. Metrics such as PixCorr, SSIM, Alex, CLIP, Eff and SwAV are commonly reported in previous reconstruction papers and should be included here for a comprehensive comparison.\n- The qualitative results in Figure 3 do not convincingly demonstrate a clear advantage of the proposed method over previous approaches. Furthermore, to substantiate the method's superiority and generalizability, the authors should provide generated results from multiple subjects.", "was_revised": true, "final_justification": "It appears there is a discrepancy between the concept of interpretability discussed in this article and its general meaning in deep learning. Emphasizing 'neurally-inspired design' over 'interpretability' in the title and introduction might be more suitable. Additionally, while the author claims the model is highly competitive, the supplementary table does not show ​​noticeable improvement​​ compared to Mindeye2 and MindBridge across all the metrics.", "rebuttal": "We respectfully but firmly disagree with the reviewer’s first comment, as the concerns raised have already been carefully analyzed and addressed in the original manuscript. We believe these issues may stem from a possible misunderstanding or a lack of thorough examination of our work. Below, we provide point-by-point responses to clarify and reinforce the corresponding aspects of our study.  We have addressed the primary concerns by first clarifying that our model's \"interpretability\" is rooted in its neuro-inspired design, and we have provided the relevant citations to ground our architecture in established neuroscience principles. Secondly, we have included a comprehensive quantitative comparison with prior work across all the requested metrics, demonstrating our model's highly competitive performance. Finally, we have presented extensive new quantitative results to substantiate our method's superior generalizability across subjects. We believe these significant additions and clarifications have thoroughly addressed the initial weaknesses, and we respectfully hope the reviewer might reconsider their evaluation in light of this substantially strengthened work.\n\n## 1. Interpretability concern:\n\nOur claim is not that MoRE-Brain is a precise, biologically detailed replica of the visual cortex, but rather that it is a *neuro-inspired computational framework* whose architectural principles are drawn from, and analogous to, well-established and emerging theories in visual neuroscience (as we cited and discussed in the third paragraph of Introduction). The \"interpretability\" we refer to is the mechanistic insight into our own model's functioning, which has been designed to reflect these neural principles. \n\nTo address the reviewer's concern, we are happy to summarize the literature that supports the core concepts behind our designs:\n\n- Hierarchical and Specialized Processing: A foundational concept in visual neuroscience is that the visual pathway is not a single, monolithic processor but a complex system involving multiple hierarchical processing stages across numerous specialized cortical areas [1, 2, 7, 8]. Our hierarchical Mixture-of-Experts (MoE) architecture is a direct embodiment of this principle, assigning specialized \"expert\" networks to process signals from distinct, functionally related voxel groups.\n\n- Dynamic Integration of Specialized Information: The brain dynamically integrates information from these specialized areas (e.g., processing form, color, motion [3]) to form a unified, coherent perception. Our Space Router is a computational mechanism designed to model this integration process, learning to route information between our specialized expert networks.\n\n- Coarse-to-Fine Temporal Dynamics: There is strong evidence that visual processing unfolds in a coarse-to-fine manner, where the global layout of a scene is processed before finer details emerge [4]. Our Time Router is explicitly designed to reflect this dynamic.\n\n- Subject-Specific Functional Topography: Finally, our approach to cross-subject generalization is motivated by findings that inter-individual brain variability often arises from spatial differences in functional network topography rather than fundamental computational changes [5, 6]. This directly inspires our strategy of sharing the core \"expert\" networks across subjects while fine-tuning only the subject-specific \"routers\" that map an individual's unique brain topography to those experts.\n\nCrucially, the value of this neuro-inspired design is validated by the fact that the trained model exhibits internal mechanisms and specializations that are remarkably consistent with established neuroscientific findings. Our analyses in Section 3 and Appendix F-I demonstrate several key examples, including: \n- Emergent Hierarchical Specialization (Figs. 6, 8, 21-23): The expert networks spontaneously learn to specialize in distinct semantic and visual features.\n- Learned Coarse-to-Fine Dynamics (Fig. 13): The Time Router learns to prioritize global, high-level information early in the generation process and fine-grained detail later, consistent with temporal dynamics observed in visual processing.\n- Meaningful Brain-Semantic Mapping (Fig. 7, 8, 17-19): The router assignments provide a transparent link between specific brain regions and their contribution to the final image, offering a clear and interpretable data flow.\n\nIn summary, the interpretability of MoRE-Brain lies in its ability to bridge the gap between abstract fMRI signals and visual reconstruction through a structured, neuro-inspired computational path. We believe this is a crucial step towards building brain decoders that are not only powerful but also scientifically insightful.\n\n[1] Alyssa A Brewer, William A Press, Nikos K Logothetis, and Brian A Wandell. Visual areas in macaque cortex measured using functional magnetic resonance imaging. Journal of Neuroscience, 22(23):10416–10426, 2002.\n\n[2] Daniel J Felleman and David C Van Essen. Distributed hierarchical processing in the primate cerebral cortex. Cerebral cortex (New York, NY: 1991), 1(1):1–47, 1991.\n\n[3] Melvyn A Goodale and A David Milner. Separate visual pathways for perception and action. Trends in neurosciences, 15(1):20–25, 1992\n\n[4] Rolf Skyberg, Seiji Tanabe, Hui Chen, and Jianhua Cang. Coarse-to-fine processing drives the efficient coding of natural scenes in mouse visual cortex. Cell reports, 38(13), 2022\n\n[5] Kevin M Anderson, et al. Heritability of cortical network topography. Proceedings of the National Academy of Sciences, 118(9):e2016271118, 2021.\n\n[6] Hannah S Savage, et al. Dissecting task-based fmri activity using normative modelling: an application to the emotional face matching task. Communications Biology, 7(1):888, 2024.\n\n[7] Kruger N, Janssen P, Kalkan S, et al. Deep hierarchies in the primate visual cortex: What can we learn for computer vision?[J]. IEEE transactions on pattern analysis and machine intelligence, 2012, 35(8): 1847-1871.\n\n[8] Tschechne S, Neumann H. Hierarchical representation of shapes in visual cortex—from localized features to figural shape segregation[J]. Frontiers in computational neuroscience, 2014, 8: 93.\n\n\n## 2. Comprehensive quantitative comparison:\n\nWe thank the reviewer for pointing out this. We chose DreamSim, SSIM, InceptionV3, and CLIP cosine similarity as metrics as they cover the low-level and high-level perception. To provide a comprehensive comparison, we here present the results of the proposed MoRE-Brain (unrefined) against baselines, averaged across the 8 subjects. We will include this result in the final version of the paper.\n|            | PixCorr↑ | SSIM↑ | Alex (2) ↑ | Alex (5) ↑ | Incep↑ |  CLIP↑ |  Eff↓ | SwAV↓ | DreamSim↓ |\n|:----------:|:-------:|:----:|:--------:|:--------:|:-----:|:-----:|:----:|:----:|:--------:|\n|  MindEye2  |   **0.282**  | 0.420 |   93.2%  |   **98.5%**  | 94.4% | 95.1% | **0.633** | **0.323** |   0.524   |\n| MindBridge |   0.163  | 0.372 |   91.4%  |   95.7%  | 92.4% | 94.8% | 0.684 | 0.426 |   0.532   |\n| MoRE-Brain |   0.176  | **0.428** |   **93.5%**  |   96.7%  | **95.6%** | **95.5%** | 0.642 | 0.345 |  **0.511**   |\n\n## 3. Qualitative results and generalizability concern:\n\nWe thank the reviewer’s perspective. In Figure 3, the proposed method produces more accurate reconstructions of the finer details (e.g., MoRE-Brain is the only model that renders the text of the “STOP” sign) and the overall environment (e.g., the kitchen). We will explicitly state these comparisons in the figure's caption to make the advantages of our method more apparent.\n\nFor the generalizability, while the rebuttal format prevents us from including new figures, we have already taken steps to address this concern and will include a more comprehensive analysis in the final version of the paper. As we detailed in Appendix C, we have already performed a quantitative analysis of MoRE-Brain's generalization capabilities. This analysis shows that our method maintains a high level of performance when trained on subject 1 and tested on others, even with limited data. \n\nTo better compare generalizability, we compare the methods’ performances when generalizing to subjects 2, 5, and 7 after being trained on subject 1. Note that for MindEye2, we fine-tune all its parameters; for MindBridge, we fine-tune it using its proposed reset-tuning method; for MoRE-Brain, we only fine-tune the routers while keeping the experts frozen. Below are the results.\n\n1 Session of Data:\n|            | SSIM↑ | InceptionV3↑ | CLIP CosineSim↑ | DreamSim↓ |\n|:----------:|:----:|:-----------:|:--------------:|:--------:|\n|  MindEye2  | **0.411** |    **68.8%**    |      0.576      |   **0.687**   |\n| MindBridge | 0.219 |    62.5%    |      0.554      |   0.804   |\n| MoRE-Brain | 0.302 |    65.3%    |      **0.703**      |   0.754   |\n\n4 Sessions of Data:\n|            | SSIM↑ | InceptionV3↑ | CLIP CosineSim↑ | DreamSim↓ |\n|:----------:|:----:|:-----------:|:--------------:|:--------:|\n|  MindEye2  | **0.409** |    81.1%    |      0.632      |  **0.624**  |\n| MindBridge | 0.262 |    71.9%    |      0.603      |   0.755   |\n| MoRE-Brain | 0.381 |    **81.5%**    |      **0.762**      |   0.639   |\n\n20 Sessions of Data:\n|            | SSIM↑ | InceptionV3↑ | CLIP CosineSim↑ | DreamSim↓ |\n|:----------:|:----:|:-----------:|:--------------:|:--------:|\n|  MindEye2  | **0.415** |    82.5%    |      0.695      |   0.587   |\n| MindBridge | 0.310 |    76.5%    |      0.717      |   0.692   |\n| MoRE-Brain | 0.384 |    **82.9%**    |      **0.804**      |   **0.586**   |\n\n40 Sessions of Data:\n|            | SSIM↑ | InceptionV3↑ | CLIP CosineSim↑ | DreamSim↓ |\n|:----------:|:----:|:-----------:|:--------------:|:--------:|\n|  MindEye2  | **0.417** |    96.5%    |      0.820      |   0.534   |\n| MindBridge | 0.375 |    94.2%    |      0.734      |   0.592   |\n| MoRE-Brain | 0.403 |    **96.8%**    |      **0.879**      |   **0.513**   |"}, {"review_id": "xnYSN8RL0p", "reviewer": "Reviewer_4LoQ", "summary": "Introduces a mixture-of-experts architecture for fMRI-to-image reconstruction. Experts can specialize decoding from specific functional regions. Time vs. Space router allows the diffusion model to be guided by the fMRI signal in a more interpretable manner. Cross-subject decoding is achievable by simple finetuning of the routers while the experts are frozen.", "questions": "* the bottleneck analyses is a bit suspect to me: the claim that MoRE-Brain is notably more sensitive to the amount of information allowed through the bottleneck does not seem to be strongly supported by the data. For all the models, the bottleneck level barely changes model performance, while MoRE-Brain is simply worse at most bottleneck levels except the last one. Any explanation as to why this is the case?\n* how well does the generalization ability compare to other models?\n* how well does does the model generalize across subjects if the experts were unfrozen?\n* can you add qualitative results for the cross-subject decoding?\n* can you present the quantitative benchmarks in a table rather than in a graph in the bottleneck analysis?", "limitations": "no discussion of societal impacts", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n* unique MoE approach\n* good interpretability results\n\nWeaknesses:\n* bottleneck analyses isn't particularly convincing, minimal quantitative comparisons to prior work\n* generalization experiments aren't particularly comprehensive (see questions)", "was_revised": true, "final_justification": "increased due to additional generalization experiments and qualitative results as requested, improving the quality of the paper slightly.", "rebuttal": "We sincerely thank the reviewer for their detailed feedback. In response, we have provided extensive new data and clarifications to address the concerns raised. We have revised our bottleneck analysis to better explain why the constraint disproportionately affects our MoE model's specialized expert representations. Furthermore, we now include comprehensive tables comparing our model's generalization ability against baselines, including an ablation with unfrozen experts, which demonstrates our method's superior performance in low-data regimes and competitive scaling. While formatting limits preclude figures in this rebuttal, we have included the requested quantitative benchmarks in table format for the bottleneck analysis and will add the suggested qualitative cross-subject results and a discussion on societal impact to the final paper. We hope these additions and clarifications have addressed your primary concerns and demonstrated the significance and strength of our work.\n\n\n## 1. Concern about bottleneck analyses:\n\nWe thank the reviewer for their insightful feedback. First, we would like to clarify that the performance of models exhibits an inclination to drop with the decrease in bottleneck size. More importantly, compared to without a bottleneck, models (especially MoRE-Brain) have a more substantial degradation in performance after adding a bottleneck. The observation that MoRE-Brain's absolute performance is lower than baselines at most bottleneck levels, yet superior without a bottleneck, is a crucial point that we would like to clarify. \n\nWe argue that this is because the bottleneck acts as a stronger constraint for MoRE-Brain’s MoE architecture, whose underlying hypothesis is that different experts specialize in capturing distinct aspects of the complex fMRI signal (as we demonstrated in the analyses of section 3 and Appendix I-J). This allows MoRE-Brain to learn a more diverse and comprehensive representation of the neural data compared to monolithic models. When no bottleneck is present, MoRE-Brain leverages this rich representation to achieve high-fidelity reconstruction, as evidenced by its superior results in the \"no bottleneck\" condition.\nHowever, the bottleneck imposes a significant constraint: all the information learned by the model must be passed through a single, low-dimensional linear projection. We hypothesize that this \"one-size-fits-all\" bottleneck is particularly detrimental to a model like MoRE-Brain due to the reduced information diversity. The bottleneck forces the specialized representations for multiple experts through a single, narrow channel creates a more severe information-loss problem for MoRE-Brain than for models that learn a single, less diverse, representation to begin with. \n\nConversely, the baseline models, which we argue rely more on powerful generative priors, are less affected by the bottleneck. Their performance is more robust because the information from the fMRI signal is less critical to their final output to begin with. Their flatter performance curves are indicative of this lower reliance on the neural data.\n\n## 2. Compare generalization ability with baseline models and when experts are unfrozen:\n\nWe compare the models’ average performances when generalizing to subject 2, 5, and 7 after trained on subject 1. Below are the results. Note that for MindEye2, we fine-tune all its parameters; for MindBridge, we fine-tune it using its proposed reset-tuning method; for MoRE-Brain, we only fine-tune the routers while keeping the experts frozen. We also test MoRE-Brain’s performance when fine-tuning all the parameters (keep experts unfrozen, MoRE-Brain (all)). Below are the results.\n\nOur proposed strategy achieves performance that is remarkably close to full fine-tuning across all data regimes, especially on key metrics like InceptionV3 and DreamSim.\n- In low-data regimes (1-4 sessions), our method significantly outperforms MindBridge and achieves strong results against MindEye2, notably scoring much higher on the CLIP Cosine Similarity metric, which measures semantic correctness.\n- As more data becomes available (20-40 sessions), our router-only approach continues to scale effectively, closing the gap and eventually matching or exceeding the performance of fully fine-tuned baselines across most metrics.\n\n\n1 Session Data\n|            | SSIM↑ | InceptionV3↑ | CLIP CosineSim↑ | DreamSim↓ |\n|:----------:|:----:|:-----------:|:--------------:|:--------:|\n|  MindEye2  | **0.411** |    **68.8%**    |      0.576      |   0.687   |\n| MindBridge | 0.219 |    62.5%    |      0.554      |   0.804   |\n| MoRE-Brain | 0.302 |    65.3%    |      0.703      |   0.754   |\n| MoRE-Brain (all) | 0.32 |    67.2%    |      **0.731**      |   **0.662**   |\n\n4 Sessions Data\n|            | SSIM↑ | InceptionV3↑ | CLIP CosineSim↑ | DreamSim↓ |\n|:----------:|:----:|:-----------:|:--------------:|:--------:|\n|  MindEye2  | **0.409** |    81.1%    |      0.632      |  0.624  |\n| MindBridge | 0.262 |    71.9%    |      0.603      |   0.755   |\n| MoRE-Brain | 0.381 |    81.5%    |      0.762      |   0.639   |\n| MoRE-Brain (all)  | 0.384 |    **82.0%**    |      **0.780**      |   **0.622** |\n\n20 Sessions Data\n|            | SSIM↑ | InceptionV3↑ | CLIP CosineSim↑ | DreamSim↓ |\n|:----------:|:----:|:-----------:|:--------------:|:--------:|\n|  MindEye2  | **0.415** |    82.5%    |      0.695      |   0.587   |\n| MindBridge | 0.310 |    76.5%    |      0.717      |   0.692   |\n| MoRE-Brain | 0.384 |    82.9%    |      0.804      |   0.586   |\n| MoRE-Brain (all)  | 0.397 |    **83.1%**    |      **0.814**      |   **0.552**   |\n\n40 Sessions Data (Full):\n|            | SSIM↑ | InceptionV3↑ | CLIP CosineSim↑ | DreamSim↓ |\n|:----------:|:----:|:-----------:|:--------------:|:--------:|\n|  MindEye2  | **0.417** |    96.5%    |      0.820      |   0.534   |\n| MindBridge | 0.375 |    94.2%    |      0.734      |   0.592   |\n| MoRE-Brain | 0.403 |    96.8%    |      0.879      |   0.513   |\n| MoRE-Brain (all)  | **0.417** |    **97.1%**    |      **0.884**  |  **0.503**   |\n\n## 3. Qualitative results on cross-subject decoding:\n\nWe are grateful for your suggestion. Due to the formatting restrictions of the rebuttal, we cannot directly add figures here. More qualitative results for each subject will be available in the final version of the paper.\n\n## 4. Quantitative benchmark as a table:\n\nThank you for your comment. Below are the qualitative results of the bottleneck analysis.\n\nDreamSim:\n|                        | Random |  B16  |  B32  |  B64  |  B128 |  B256 | No Bottleneck | Ceiling |\n|:----------------------:|:------:|:-----:|:-----:|:-----:|:-----:|:-----:|:-------------:|:-------:|\n|        MindEye2        |  0.831 | 0.570 | 0.567 | 0.561 | 0.569 | 0.547 |     0.516     |  0.368  |\n|       MindBridge       |  0.795 | 0.599 | 0.595 | 0.571 | 0.542 | 0.545 |     0.531     |  0.305  |\n| MoRE-Brain (unrefined) |  0.893 | 0.681 | 0.674 | 0.645 | 0.651 | 0.634 |     0.518     |  0.374  |\n|       MoRE-Brain       |  0.893 | 0.667 | 0.651 | 0.639 | 0.625 | 0.621 |     0.501     |  0.374  |\n\nSSIM:\n|                        | Random |  B16 |  B32 |  B64 | B128 | B256 | No Bottleneck | Ceiling |\n|:----------------------:|:------:|:----:|:----:|:----:|:----:|:----:|:-------------:|:-------:|\n|        MindEye2        |  0.308  | 0.389 | 0.405 | 0.416 | 0.422 | 0.418 |      0.420     |   0.528  |\n|       MindBridge       |  0.318  | 0.359 | 0.363 | 0.353 | 0.372 | 0.369 |      0.372     |   0.455  |\n| MoRE-Brain (unrefined) |  0.204  | 0.318 | 0.341 | 0.352 | 0.365 | 0.368 |      0.428     |   0.512  |\n|       MoRE-Brain       |  0.204  | 0.319 | 0.345 | 0.365 | 0.367 | 0.379 |      0.435     |   0.512  |\n\nInceptionV3:\n|                        | Random |  B16  |  B32  |  B64  |  B128 |  B256 | No Bottleneck | Ceiling |\n|:----------------------:|:------:|:-----:|:-----:|:-----:|:-----:|:-----:|:-------------:|:-------:|\n|        MindEye2        |  50.4% | 87.1% | 88.6% | 90.7% | 88.5% | 91.1% |     94.4%     |  99.5%  |\n|       MindBridge       |  50.9% | 85.3% | 87.2% | 90.4% | 92.1% | 93.0% |     92.4%     |  99.6%  |\n| MoRE-Brain (unrefined) |  45.9% | 80.1% | 81.1% | 84.7% | 83.4% | 86.1% |     95.6%     |  99.3%  |\n|       MoRE-Brain       |  45.9% | 83.2% | 85.1% | 86.1% | 87.4% | 90.1% |     96.0%     |  99.3%  |\n\n\nCLIP cosine similarity:\n|            | Random |  B16  |  B32  |  B64  |  B128 |  B256 | No Bottleneck | Ceiling |\n|:----------:|:------:|:-----:|:-----:|:-----:|:-----:|:-----:|:-------------:|:-------:|\n|  MindEye2  |  0.801 | 0.819 | 0.819 | 0.821 | 0.822 | 0.820 |     0.821     |    1    |\n| MindBridge |  0.386 | 0.153 | 0.177 | 0.401 | 0.481 | 0.500 |     0.585     |    1    |\n| MoRE-Brain |  0.606 | 0.675 | 0.681 | 0.711 | 0.749 | 0.784 |     0.811     |    1    |\n\n## 5. Discussion on societal impact:\n\nWe acknowledge the reviewer's comment and agree that a discussion of societal impacts is important. Our research in fMRI-based visual decoding, as part of the broader field of Brain-Computer Interfaces (BCIs), has significant potential for positive societal impact, particularly in developing advanced communication aids for individuals with severe motor disabilities [1]. By enabling the direct translation of brain activity into signals for controlling external devices, our work could significantly improve the quality of life and autonomy for these individuals. We will include this discuss in the final version of the paper. \n\n[1] Lee K Y, Jang D. Ethical and social issues behind brain-computer interface[C]//2013 International Winter Workshop on Brain-Computer Interface (BCI). IEEE, 2013: 72-75."}, {"review_id": "VppSnOBwp4", "reviewer": "Reviewer_HRWS", "summary": "This paper introduces MoRE-Brain, a novel framework for decoding visual stimuli from fMRI data. The core of the method is a neuro-inspired hierarchical Mixture-of-Experts (MoE) architecture, which mimics the functional specialization and hierarchical processing of the human visual system.\n\nThe framework operates in two stages: first, an MoE-based encoder is trained to map fMRI signals from distinct voxel groups to CLIP embeddings. Second, SDXL is used to generate images, guided by the fMRI embeddings via a novel dual-stage (Time and Space) routing mechanism.\n\nThe authors claim three main contributions: 1) a new brain-inspired MoE architecture for neuro-decoding, 2) efficient cross-subject generalization by adapting only subject-specific routers while sharing expert networks, and 3) enhanced interpretability by revealing how different modeled brain regions contribute to the reconstruction. The method is evaluated on the Natural Scenes Dataset (NSD), where it shows competitive performance and demonstrates strong results in generalization and interpretability analyses.", "questions": "Please see the points raised in the Weaknesses section above.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "# Strength\n1. Novel and Well-Motivated Architecture: Using a hierarchical Mixture-of-Experts (MoE) architecture is well-motivated, as it is based on principles of neuroscience, specifically the concepts of functional specialization and hierarchical processing in the visual cortex. This represents a principled departure from more monolithic decoder architectures and provides a solid foundation for the paper's interpretability claims.\n\n2. Practical Generalization Framework: The approach of freezing the shared \"expert\" networks and fine-tuning only the subject-specific \"routers\" is an elegant and practical way to address cross-subject generalization. The experiments, which show good performance on new subjects with only a small fraction of their data (e.g., 1-4 sessions), effectively validate this approach.\n\n3. Extensive and Insightful Interpretability Analysis: A major highlight of this work is its deep dive into model interpretability, which goes far beyond what is typical for reconstruction papers in this field. The authors use a combination of techniques, including ICA on GradientSHAP attributions, to link model components to known brain networks. The analyses successfully demonstrate emergent functional specialization, showing that experts learn to focus on different spatial regions and develop preferences for specific semantic categories at higher hierarchical levels. This provides compelling evidence that the model is not just a black box but learns neurophysiologically plausible mappings.\n\n4. Rigorous Evaluation via Bottleneck Analysis: The inclusion of a bottleneck analysis to probe the model's reliance on fMRI signals versus generative priors is commendable. The results, which show that MoRE-Brain's performance degrades more significantly with restricted fMRI information compared to baselines, lend strong support to the claim that the model genuinely utilizes the neural data rather than over-relying on the priors of the diffusion model.\n\n\n# Weakness\n\n1. The presentation quality needs attention. The architectural diagrams (Figures 1 and 2) have distracting backgrounds that reduce clarity. And in Line 194, Figure 4 does not show numeric results as \"quantitatively\" states, a table the exact numerical values should be more precise.\n\n2. Insufficient Detail on Model Efficiency: one of the claim is the efficiency of cross-subject generalization. However, the paper fails to quantify this efficiency properly. It never states the number of tunable parameters in the fMRI routers relative to the total number of parameters in the model. Without this detail, it is hard to gauge the actual computational savings of the proposed fine-tuning strategy.", "was_revised": true, "final_justification": "The authors' rebuttal addresses my concerns, so I raise my rating to 5.", "rebuttal": "We sincerely thank the reviewer for their valuable feedback and positive assessment of our work's strengths. We have diligently addressed the identified weaknesses. The architectural diagrams have been revised for better clarity, a table with precise numerical values has been added as requested, and we have now quantified the efficiency of our model by specifying the number of tunable parameters. We hope these revisions fully address the reviewer's concerns and have strengthened the manuscript. Below are the detailed response to each comment.\n\n## 1. Presentation quality concern:\n\nWe thank the reviewer for the valuable feedback. In the revised version, we will remove the distracting backgrounds from these figures to enhance their clarity. Additionally, as you suggested, we added tables with exact numeric results to accompany Figure 4 in the final version of the paper. The results are quoted below.\n\nDreamSim:\n|                        | Random |  B16  |  B32  |  B64  |  B128 |  B256 | No Bottleneck | Ceiling |\n|:----------------------:|:------:|:-----:|:-----:|:-----:|:-----:|:-----:|:-------------:|:-------:|\n|        MindEye2        |  0.831 | 0.570 | 0.567 | 0.561 | 0.569 | 0.547 |     0.516     |  0.368  |\n|       MindBridge       |  0.795 | 0.599 | 0.595 | 0.571 | 0.542 | 0.545 |     0.531     |  0.305  |\n| MoRE-Brain (unrefined) |  0.893 | 0.681 | 0.674 | 0.645 | 0.651 | 0.634 |     0.518     |  0.374  |\n|       MoRE-Brain       |  0.893 | 0.667 | 0.651 | 0.639 | 0.625 | 0.621 |     0.501     |  0.374  |\n\n\nSSIM:\n|                        | Random |  B16 |  B32 |  B64 | B128 | B256 | No Bottleneck | Ceiling |\n|:----------------------:|:------:|:----:|:----:|:----:|:----:|:----:|:-------------:|:-------:|\n|        MindEye2        |  0.308  | 0.389 | 0.405 | 0.416 | 0.422 | 0.418 |      0.420     |   0.528  |\n|       MindBridge       |  0.318  | 0.359 | 0.363 | 0.353 | 0.372 | 0.369 |      0.372     |   0.455  |\n| MoRE-Brain (unrefined) |  0.204  | 0.318 | 0.341 | 0.352 | 0.365 | 0.368 |      0.428     |   0.512  |\n|       MoRE-Brain       |  0.204  | 0.319 | 0.345 | 0.365 | 0.367 | 0.379 |      0.435     |   0.512  |\n\n\nInceptionV3:\n|                        | Random |  B16  |  B32  |  B64  |  B128 |  B256 | No Bottleneck | Ceiling |\n|:----------------------:|:------:|:-----:|:-----:|:-----:|:-----:|:-----:|:-------------:|:-------:|\n|        MindEye2        |  50.4% | 87.1% | 88.6% | 90.7% | 88.5% | 91.1% |     94.4%     |  99.5%  |\n|       MindBridge       |  50.9% | 85.3% | 87.2% | 90.4% | 92.1% | 93.0% |     92.4%     |  99.6%  |\n| MoRE-Brain (unrefined) |  45.9% | 80.1% | 81.1% | 84.7% | 83.4% | 86.1% |     95.6%     |  99.3%  |\n|       MoRE-Brain       |  45.9% | 83.2% | 85.1% | 86.1% | 87.4% | 90.1% |     96.0%     |  99.3%  |\n\n\nCLIP cosine similarity:\n|            | Random |  B16  |  B32  |  B64  |  B128 |  B256 | No Bottleneck | Ceiling |\n|:----------:|:------:|:-----:|:-----:|:-----:|:-----:|:-----:|:-------------:|:-------:|\n|  MindEye2  |  0.801 | 0.819 | 0.819 | 0.821 | 0.822 | 0.820 |     0.821     |    1    |\n| MindBridge |  0.386 | 0.153 | 0.177 | 0.401 | 0.481 | 0.500 |     0.585     |    1    |\n| MoRE-Brain |  0.606 | 0.675 | 0.681 | 0.711 | 0.749 | 0.784 |     0.811     |    1    |\n\n## 2. Insufficient Detail on Model Efficiency:\n\nThank you very much for the constructive feedback. We agree that our claim regarding the efficiency of cross-subject generalization needs to be substantiated with a quantitative analysis of the model's parameters.\n\nTo address your concern, we count the number of parameters of MoRE-Brain and compare it with MindEye2 and MindBridge. We also compute the percentage of the trainable parameters during fine-tuning. Note that MindEye2 does not have a special fine-tuning strategy, and MindBridge proposes a reset-tuning strategy, which freezes the “translator” module.\n\nFor initial training, MoRE-Brain is substantially more lightweight than competing methods. The table below shows that our model has less than half the parameters of MindEye2 and trains a smaller percentage of its parameters compared to the fine-tuning strategy proposed for MindBridge. We have updated the manuscript and added the results in the appendix to reflect this analysis.\n|            | Total Params (M) | Trainable (%)|\n|:----------:|:----------:|:---------:|\n|  MindEye2  |    729.3   |     100%    |\n| MindBridge |    552.9   |   98.48%  |\n| MoRE-Brain |    293.4   |   44.84%  |"}, {"review_id": "BpNaxz6TSW", "reviewer": "Reviewer_92RU", "summary": "This paper introduces MoRE-Brain, a new model for reconstructing images from fMRI data. Instead of using a single model for the whole brain, it uses multiple smaller expert models, each assigned to different brain regions. A routing system decides which experts to use across time and space, which makes the model more interpretable and flexible. The authors provide detailed visualizations and analysis to connect model components with brain function.", "questions": "I noticed that the visual reconstructions from MoRE-Brain look better than MindEye2, but the quantitative metrics do not reflect this. Could the authors comment on why this discrepancy might exist?", "limitations": "refer to weakness", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe paper presents two novel and biologically inspired ideas. First, a hierarchical mixture-of-experts (MoE) structure that mirrors the visual hierarchy in the brain. Second, a dynamic time and space routing mechanism that could relate to how the brain integrates visual information over time and across regions. The design is not only technically interesting but also offers interpretability, which is rare in brain decoding models.\n\nThe authors provide rich visual analysis to support the interpretability of their model, especially linking components to brain functions. \n\nThe paper is well written and clearly organized.\n\nWeaknesses:\n\nWhile the paper discusses the challenge of cross-subject variability (Lines 42–45: “Due to inter-individual brain variability, models often require subject-specific training. While approaches like mapping subject data to a shared space or utilizing subject-specific tokens exist, they typically necessitate extra training and are difficult to generalize to new subjects.”), the proposed method still requires fine-tuning on each new subject, either partially or fully. This is similar to what methods like MindEye2 and MindBridge already do. So in practice, it still relies on partial fine-tuning for new subjects, and the issue is not entirely resolved. \n\nThe quantitative performance on standard image reconstruction metrics is not the best among competing methods.", "was_revised": true, "final_justification": "I'm positive to this paper.", "rebuttal": "Thank you for your constructive feedback. We have clarified that our contribution is a more efficient and principled framework for cross-subject adaptation, rather than a zero-shot solution. Our new results show that our lightweight router-only fine-tuning is highly effective and efficient. Furthermore, our expanded evaluation demonstrates that MoRE-Brain is highly competitive, achieving state-of-the-art results on the majority of nine standard metrics. We believe these substantial revisions and new data strongly validate our neuro-inspired approach, and they more clearly establish the significance of our contributions.\n\n\n## 1. Fine-tuning for new subjects:\n\nThank you for your careful reading and for your insightful comment regarding cross-subject generalization. We acknowledge that our method does not offer a zero-shot solution and that some form of adaptation is still necessary. Our primary contribution is not the complete elimination of subject-specific adaptation, but rather a novel framework that makes this adaptation significantly more data-efficient, computationally lightweight, and neuro-scientifically interpretable.\n\nThe key distinction between MoRE-Brain and prior methods lies in our neuro-inspired MoE architecture, which allows for a more principled approach to generalization.\n\n- Principled Separation of Components: Our model is explicitly designed to separate the core, generalizable visual decoding computations (the expert networks) from the subject-specific mappings (the router networks). This design is based on the neuroscientific hypothesis  (Lines 60-62). By fine-tuning only the lightweight routers, we are specifically adapting the model to the new subject's unique brain layout.\n\n- Demonstrated Performance and Efficiency of Router-Only Fine-Tuning: To validate this strategy, we have added experimental results comparing our proposed router-only fine-tuning against baselines and a full fine-tuning of our own model. All models are trained on subject 1 and generalize to subjects 2, 5, and 7. Here we report the average performance on these 3 subjects. These experiments yield several key insights:\n\n    - Router-only tuning is highly effective: The tables below show a direct comparison between fine-tuning only the routers (MoRE-Brain) and fine-tuning all parameters (MoRE-Brain (all)). Our proposed strategy achieves performance that is *remarkably close to full fine-tuning across all data regimes*, especially on key metrics like InceptionV3 and DreamSim.\n\n    - Competitive performance against state-of-the-art methods: Even with this efficient, partial fine-tuning, MoRE-Brain is highly competitive. In low-data regimes (1-4 sessions), our method significantly outperforms MindBridge and achieves strong results against MindEye2, notably scoring much higher on the CLIP Cosine Similarity metric, which measures semantic correctness. As more data becomes available (20-40 sessions), our router-only approach continues to scale effectively, closing the gap and eventually matching or exceeding the performance of fully fine-tuned baselines across most metrics.\n\n(MoRE-Brain uses router-only fine-tuning. Baselines are fine-tuned as per their original papers.)\n\n1 Session Data:\n|            | SSIM↑ | InceptionV3↑ | CLIP CosineSim↑ | DreamSim↓ |\n|:----------:|:----:|:-----------:|:--------------:|:--------:|\n|  MindEye2  | **0.411** |    **68.8%**    |      0.576      |   0.687   |\n| MindBridge | 0.219 |    62.5%    |      0.554      |   0.804   |\n| MoRE-Brain | 0.302 |    65.3%    |      0.703      |   0.754   |\n| MoRE-Brain (all) | 0.32 |    67.2%    |      **0.731**      |   **0.662**   |\n\n4 Sessions Data:\n|            | SSIM↑ | InceptionV3↑ | CLIP CosineSim↑ | DreamSim↓ |\n|:----------:|:----:|:-----------:|:--------------:|:--------:|\n|  MindEye2  | **0.409** |    81.1%    |      0.632      |  0.624  |\n| MindBridge | 0.262 |    71.9%    |      0.603      |   0.755   |\n| MoRE-Brain | 0.381 |    81.5%    |      0.762      |   0.639   |\n| MoRE-Brain (all)  | 0.384 |    **82.0%**    |      **0.780**      |   **0.622** |\n\n20 Sessions Data:\n|            | SSIM↑ | InceptionV3↑ | CLIP CosineSim↑ | DreamSim↓ |\n|:----------:|:----:|:-----------:|:--------------:|:--------:|\n|  MindEye2  | **0.415** |    82.5%    |      0.695      |   0.587   |\n| MindBridge | 0.310 |    76.5%    |      0.717      |   0.692   |\n| MoRE-Brain | 0.384 |    82.9%    |      0.804      |   0.586   |\n| MoRE-Brain (all)  | 0.397 |    **83.1%**    |      **0.814**      |   **0.552**   |\n\n40 Sessions Data (Full):\n|            | SSIM↑ | InceptionV3↑ | CLIP CosineSim↑ | DreamSim↓ |\n|:----------:|:----:|:-----------:|:--------------:|:--------:|\n|  MindEye2  | **0.417** |    96.5%    |      0.820      |   0.534   |\n| MindBridge | 0.375 |    94.2%    |      0.734      |   0.592   |\n| MoRE-Brain | 0.403 |    96.8%    |      0.879      |   0.513   |\n| MoRE-Brain (all)  | **0.417** |    **97.1%**    |      **0.884**  |  **0.503**   |\n\n- Superior Parameter Efficiency: This competitive performance is achieved with a fraction of the trainable parameters. As shown below, our router-only fine-tuning approach is substantially more lightweight than competing methods, requiring updates to only **44.8%** of the model's parameters, which are themselves less than half the total parameters of MindEye2.\n\n|            | Total Params (M) | Trainable (%)|\n|:----------:|:----------:|:---------:|\n|  MindEye2  |    729.3   |     100%     |\n| MindBridge |    552.9   |   98.48%  |\n| MoRE-Brain |    293.4   |   44.84%  |\n\nIn response to your feedback and to make these advantages clearer, we will make the following revisions to our manuscript:\n- In the Abstract and Introduction, we will refine our wording to emphasize that we are \"proposing a more effective and efficient framework for cross-subject adaptation\" rather than completely \"solving\" the issue.\n- We will update Section 3.3 (Cross-Subject Generalization) to include a summary of these new comparative results, highlighting how our architectural separation enables these data and parameter efficiency gains.\n- We will add the full quantitative tables and analysis to the appendix to provide a comprehensive comparison between MoRE-Brain’s fine-tuning strategy and the baseline methods.\nWe believe these new results and the corresponding revisions provide a much stronger and clearer case for our contribution. We thank you again for pushing us to clarify this important point.\n\n## 2. Quantitative performance concern:\n\nWe thank the reviewer for their constructive feedback and for raising this important point. We acknowledge that in our original submission, the quantitative comparison may not have fully captured the comprehensive performance and efficiency of our proposed MoRE-Brain model.\n\nTo provide a clearer and more comprehensive comparison, we have added more results to compare MoRE-Brain with MindEye2 and MindBridge, across a broader set of 9 standard metrics for standard visual reconstruction. These new results will be included in the revised manuscript. Here is the summary of the results:\n\n|            | PixCorr↑ | SSIM↑ | Alex (2) ↑ | Alex (5) ↑ | Incep↑ |  CLIP↑ |  Eff↓ | SwAV↓ | DreamSim↓ |\n|:----------:|:-------:|:----:|:--------:|:--------:|:-----:|:-----:|:----:|:----:|:--------:|\n|  MindEye2  |   **0.282**  | 0.420 |   93.2%  |   **98.5%**  | 94.4% | 95.1% | **0.633** | **0.323** |   0.524   |\n| MindBridge |   0.163  | 0.372 |   91.4%  |   95.7%  | 92.4% | 94.8% | 0.684 | 0.426 |   0.532   |\n| MoRE-Brain |   0.176  | **0.428** |   **93.5%**  |   96.7%  | **95.6%** | **95.5%** | 0.642 | 0.345 |  **0.511**   |\n\nAs the table shows, MoRE-Brain achieves the best results on 5 of the 9 metrics, including SSIM, Alex(2), Inception, CLIP, and DreamSim. Furthermore, in our previous response, we also show our model’s architectural efficiency, which only has less than half of the parameters compared to MindEye2, and only trains less than 50% of parameters during fine-tuning. Moreover, our additional results comparing MoRE-Brain with other methods when generalizing to new subjects with limited data also confirm our method’s effectiveness.\n\n## 3. Visual reconstruction discrepancy:\n\nThank you for this insightful question. We agree that the qualitative superiority of MoRE-Brain’s results may not be fully captured by the quantitative metrics due to the following reasons.\n\nFirst, the numeric metrics themselves may not align with human perception. For example, we observe that MindEye2 achieves an exceptionally high score on metrics such as PixCorr, which is sensitive to low-level statistics and pixel-perfect alignment. A model can score well on such metrics by producing a technically \"correct\" but blurry or less coherent image. On the other hand, MoRE-Brain scores the highest on several high-level metrics (as in the table we provided in the previous response), such as DreamSim, which is designed to align with human perception. This suggests that MoRE-Brain can be effective at capturing semantic contents and coherent structure of the scene, while it may not be optimized for matching low-level statistics.\n\nSecond, the qualitative results in Figure 3 are randomly drawn from the test set, while MoRE-Brain may produce failed reconstructions for some samples. Due to the formality requirement of the rebuttal, we cannot directly show these reconstructions here. We will include the 5 best and 5 worst reconstructions for each method in the final paper to provide a more comprehensive comparison."}]

Author rebuttal:

Meta-review: The manuscript has been reviewed by four reviewers. 

The authors provided detailed responses to the concerns of the reviewers.

After the rebuttal, one reviewer rated acceptance, while other three gave borderlines, indicating that no reviewer is willing to (even weakly) reject the manuscript.

Despite the minor flaws, the reviewers praised the submission for its unique and novel MoE approach, good interpretability, and promising results.

The AE agrees that the manuscript should be presented to a broad audiance, and recommends acceptance. Congrats!
Decision: Accept (poster)