SynBrain: Enhancing Visual-to-fMRI Synthesis via Probabilistic Representation Learning
Authors: Weijian Mai
Jiamin Wu
Yu Zhu
Zhouheng Yao
Dongzhan Zhou
Andrew Luo
Qihao Zheng
Wanli Ouyang
Chunfeng Song
Venue: NeurIPS 2025 poster ()
Keywords: Brain encoding
Probabilistic modeling
Cross-modal generation
Primary area: neuroscience_and_cognitive_science

Abstract: Deciphering how visual stimuli are transformed into cortical responses is a fundamental challenge in computational neuroscience. This visual-to-neural mapping is inherently a one-to-many relationship, as identical visual inputs reliably evoke variable hemodynamic responses across trials, contexts, and subjects. However, existing deterministic methods struggle to simultaneously model this biological variability while capturing the underlying functional consistency that encodes stimulus information. To address these limitations, we propose SynBrain, a generative framework that simulates the transformation from visual semantics to neural responses in a probabilistic and biologically interpretable manner. SynBrain introduces two key components: (i) BrainVAE models neural representations as continuous probability distributions via probabilistic learning while maintaining functional consistency through visual semantic constraints; (ii) A Semantic-to-Neural Mapper acts as a semantic transmission pathway, projecting visual semantics into the neural response manifold to facilitate high-fidelity fMRI synthesis. Experimental results demonstrate that SynBrain surpasses state-of-the-art methods in subject-specific visual-to-fMRI encoding performance. Furthermore, SynBrain adapts efficiently to new subjects with few-shot data and synthesizes high-quality fMRI signals that are effective in improving data-limited fMRI-to-image decoding performance. Beyond that, SynBrain reveals functional consistency across trials and subjects, with synthesized signals capturing interpretable patterns shaped by biological neural variability. 
Our code is available at https://github.com/MichaelMaiii/SynBrain.

SynBrain: Enhancing Visual-to-fMRR Synthesis via Probabilistic Representation Learning
Weijian Mai
Jiamin Wu Yu Zhu',
Zhouheng Yao Dongzhan Zhou',
Andrew F. Luo². Qihao Zheng', Wanli Ouyang".,
Chunfeng Song
Shanghai Artificial Intelligence Laboratory
"The University of Hong Kong
The Chinese University of Hong Kong
1Shanghai Innovation Institute
Abstract
Deciphering how visual stimuli are transformed into cortical responses is funda- mental challenge in computational neuroscience. This visual-to-nent mapping is inherently a one-to-many relationship, as identical visual inputs reliably evoke variable hemodynamic responses across trials, contexts, and subjects. However, existing deterministic methods struggle to simultaneously model this biological variability while capturing the underlying functional consistency that encodes stimulus information. To address these limitations, we propose SynBrain, a gen- erative framework that simulates the transformation from visual semantics neural responses in probabilistic and biologically interpretable manner. Syn- Brain introduces two key components: (i) Brain VAE models neural represen- tations as continuous probability distributions via probabilistic learning while maintaining functional consistency through visual semantic constraints; (ii) A Semantic-to-Neural Mapper acts as a semantic transmission pathway, project- ing visual semantics into the neural response manifold to facilitate high-fidelity fMRI synthesis. Experimental results demonstrate that SynBrain surpasses state- of-the-art methods in subject-specific visual-to-fMRI encoding performance. Fur- thermore, SynBrain adapts efficiently to new subjects with few-shot data and synthesizes high-quality fMRI signals that are effective in improving data-limited IMRI-to-image decoding performance. Beyond that, SynBrain reveals functional consistency across trials and subjects, with synthesized signals capturing inter- pretable patterns shaped by biological neural variability. Our code is available at https://githun www./oico.acaae.cc http://ichallMMaii///y/aaeeee
1 Introduction
Understanding how the human brain transforms visual stimuli into structured patterns of neural activity remains one of the fundamental challenges in computational neuroscience [37] [3] 24 [8] [54). This task, commonly referred to as brain visual encoding, seeks model the functional mapping from external visual perception to spatially distributed neural responses across the cortex, uncovering how high-level visual semantics are represented neural populations 31 332 4 In recent years, functional magnetic resonance imaging (fMRI) has emerged as dominant modality for brain encoding. As a noninvasive neuroimaging technique, fMRI measures blood-oxygen wenyeen-liee www.oeeeeeeeeelooooeoeeee -eee--deended signals as indirect proxies for neuronal activity with high spatial resolution [38). By translating visual inputs into fMRI- measured neural patterns, brain encoding models not only advance our understanding human visual perception but also lay the groundwork for applications in neural decoding
35
g 47
49
28
34 60),
cognitive modeling 63 66]. and brain-machin machine interfaces [3] 26
"This work was done during his internship Shanghai Artificial Intelligence Laboratory. Corresponding Author (wujiaminl @pjlab.org.en)
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

[-MRI Synthesis
Brain Functional Analysis
Source-Subject Full-Data Training
MmM S Source
Syn Brain
Saneet
Snower
Mmw
MMM +
Mmn MWW o00
Visual Stimulus
Novel-Subject
Raw-IMRI
Syuthesize-IIMR
Few-shot Adaptation
Snovel
Data Augmentation
Figure I: Overview of SynBrain for subject-adaptive visual-to-IMRI synthesis and downstream decoding applications. SynBrain is trained full fMRI recordings from source subject and adapted to novel subjects using limited data. It generates semantically consistent neural responses that support brain functional analysis and enhance downstream decoding through synthetic data augmentation.
Recent advances in brain encoding have predominantly adopted either gggression-based or de- terministic generative strategies map visual stimuli or their semantic representations to corre- sponding fMRI recordings. Regression models typically learn a deterministic function that directly predicts voxel-level brain activity from visual stimuli using linear regression or deep neural net- works [6, [ [33 (2, 5 3 32). Differently, generative brain encoding methods such as MindSim- ulator 4 reformulate brain encoding as visual-to-fMin MMMIII synthesis process conditioned on visual input. MindSimulator uses deterministic AutoEncoder to reconstruct brain activity from semantic features, aligning fMRI latent space to visual stimuli via diffusion- based semantic-to-later mapping.
However, such deterministic strategies in existing methods fail to model the inherent one-to-man -any mapping nature of neural signals. Specifically, existing large-scale neuroimaging studies (23 indicate that repeated presentations of the same visual stimulus can elicit notably different fMRI responses across trials and subjects. This highlights a fundamental characteristic of brain activity: the mapping from visual stimuli to neural responses is inherently one-to-many, influenced by trial-level noise, brain attentional fluctuations, and inter-individual variability. Given this neural property, existing methods face three key limitations. (1) Deterministic neural modeling: previous deterministic models struggle with the one-to-many nature of brain responses, as they produce a unique latent representation per input and often collapse diverse neural patterns into non-informative averaged response that lacks semantic or physiological validity. Although MindSimulator's 's diffusion- based latent mapping attempts to introduce variability through stochastic sampling, its core fMRI synthesis module relies on a deterministic AutoEncoder without probabilistic modeling. (2) Lack of anactionally-coniitern
variability: current approaches fail to model neural responses that are simultaneously variable in their specific patterns yet consistent their functional encoding of stimulus information. (3) Limited utility synthetic training data: the lack of cross-subjes transfcrability restricts their applicability as augmentation sources for neural decoding in data-limited scenarios. These challenges motivate a key research question: how can effectively model neural response distributions to visual stimuli with functional consistency?
To address these limitations, we propose SynBrain. generative framework that models fMRI responses as a semannic-conditionne,, continuous probability distribution. SynBrain is designed to simulate the transformation from visual semantics to brain activity in a manner that is probabilistic and biologically interpretable. At the core of SynBrain is BrainVAE. a well-designed variational model that incorporates convolutional feature extractors and attention modules to mitigate unstable training dynamics observed in MLP-based variational architectures. Unlike deterministic autoencoders, VAE Icarns probabilistic latent space of fMRI responses explicitly conditioned high- visual semantics, allowing it to generate diverse 1MRI signals that reflect neural variability while preserving functional consistency. Moreover, we incorporate lightweight Transformer module that establishes semantic embeddings to the probabilistic latent space of Brain VAE. S2N Mapper achieves one- step semantic-tor ne-urall mapping for stable and semantically coherent fMRI synthesis, avoiding the distribution mismatch problems observed in previous multi-step diffusion-basee methods 4..
Comaantic-o-NNuure (S2N) Mapper, cointto-diistrbutiin mapping from fixed CLIP
Through this architecture, SynBrain facilitates a biologically grounded probabilistic framework for visual-to-IMRI synthesis, effectively balancing voxel-level neural fidelity with semantic consistency.

---

S1: BrainVAE
S2: S2N Mapper
S3: Visual-to-MMRL Synthesix
MMw
Zeup
Zalion
Fnodder
CLIP
SZN Mapper
Danodaa
Brain Decoder
Agg
Mapper
Vaual
One
Synthesis
FMRI
Acup
Visualia Coniaiiiationopptiononlllll
Brain Encodkr
CLIP
Brain Encoder
+ Mmm
Mmm +
IMRI-to-Image Generator B
MMM F-RII
Synthasized 1.MRI
[-RR
Visual Stimuli
Figure 2: Overview of the SynBrain framework. Stage 1: Brain VAE models the probabilistic distribution of IMRI responses conditioned on CLIP visual embeddings ECLIP; Stage 2: S2N Mapper learns to map #CLIP into the latent space of BrainVAEE Stage 3: At inference, the frozen S2N Mapper performs one-step mapping from ZCLIP the BrainVAB latent space for visual- to-fMRI synthesis. Synthesized fMRI could be further visualized via pretrained fMRI-to-image generator.
Specifically, its advantages are reflected in two aspects: 1) Superior subject-specific encoding performance: through probabilistic modeling of trial-level functional consistency, SynBrain signifi- cantly outperforms prior methods in subject-specific visual-to-IMRI synthesis. 2) Effective few-shot adaptation to novel subjects and utility as synthetic training data: SynBrain demonstrates strong generalization unseen subjects data-limited settings. By disentangling subject-specific variability from semantic representations, it transfers the synthesis capability for new subjects with minimal supervision while maintaining functional alignment. Furthermore, the high-fidelity synthesized signals can serve as effective data augmentations for downstream IMRI-to-image decoding under data-limited conditions, as shown in Figure T Our contributions are as follows:
We propose SynBrain, generative framework for iisual-to-MMRR synthesis that models neural responses as semantic-conditiones probability distributions. By integrating proba- bilistic learning, SynBrain captures the one-to-many nature neural responses to better reflect biological variability and synthesize semantically consistent neural responses.
SynBrain advances visual-to-fMRI synthesis by achieving superior subject-specific and few-shot novel-subject adaptation performance. Moreover, its high-quality synthesized signals prove effective as data augmentations for improving FMRI-to-image decoding in data-scarcity scenarios.
SynBrain reveals cross-trial and cross-subject functional consistency through brain func- tional analysis, demonstrating that the synthesized signals preserve interpretable cortical patterns reflecting underlying biological neural variability.
2 Methodology
The fundamental goal of visual-to-fMRR synthesis is to generate brain activity patterns that faithfully reflect the semantics of visual stimuli while capturing the inherent variability observed biological neural responses. However, most existing approaches rely on deterministic mappings from vision to fMRI, producing a single fixed output for each input. These methods fail to model the one-to- many nature of brain responses, which is influenced by trial-level noise, attentional fluctuation, and sewviduul-speecffe neural patterns.
To address this issue, we propose SynBrain, generative framework that synthesizes fMRI responses from visual stimuli by modeling the semantic-conditioned distribution of neural activity. SynBrain consists of two key components: 1) Brain VAE, variational backbone for semantic-guian variational

---

MLP-AE
BrounWAE
MINK Loss
CLIP Loss
MLP Blocks
Convid
30000
37000
MILP-AM ALE
MIRAVAE
MIP-AE
Interpolate
36000-
BrainVAE
Desemministic Mapping
Up-Sumple
34080
35000
Laws
BrainVAE
Blocks
MLP Blocks
Attention
ML.P
MLP-VAE
Egoch
Epach
Variational Samppling
MLP Blocks
Image Retrieval Accuracy
Brain Retricval Accurscy
1.0
MIPPAF
MIP
0.8-
0.5
MAP-VAE
Varistional Sampling
Atlention
Accurncy
ReviV
Down-Sample
0.4
MLP AB
0.4
MLP
Ada Maxpoll
MLP--VA
BrainVAR
0.2-
Blocks
Convld
10
20
30
40
0.9
20
30
4n
Epoch
Epach
Figure Architecture and performance comparison of MLP-based baselines and our proposed Brain VAE. Left: Architecture comparisons; Right: Validation performance comparisons.
modeling of neural response distributions, and 2) S2N Mapper, semantic-to-neura projection module. The overall framework is trained two stages and enables visual-to-fMil fo-MMRI synthesis during inference with optional IMRI-to-image visualization procedure using the MindEye2 [49] generator, as shown in Figure
2.1 BrainVAE
Human brain responses to visual stimuli are inherently variable across trials and subjects, yet preserve consistent semantic structure [2T] 24]. Traditional encoding models often treat this mapping as deterministic, ignoring both within-subject and inter-subject variability. To address this, Brain VAE models fMRI signals as samples from a conditionally structured latent distribution, conditioned on the high-level semantic embedding derived from the visual input.
Motivation and Architecture Comparison. MindSimulator 4 has employed MLP-based au- toencoders (MLP-AE) to map high-dimensional fMRI signals into a deterministic latent space. To explore the benefits of variational modeling, we first incorporated a variational formulation into this architecture, forming bascline termed MLP-VAE. However, we found that MLP- VAE suffers from unstable training. with divergence in MSE loss (Figure This might be attributed to MLP's lack of spatial inductive bias and loken-wise independence. To tackle this issue, we propose Brain VAE, a well-designed variational architecture for stable and effective fMRI representation learning. As shown in Figure 3] Left, Brain VAE integrates convolutional layers extract local voxel features and attention layers to capture long-range inter-voxel dependencies, leading to a smoother and more expressive latent space for enhanced fMRI synthesis. 3 demonstrates that Brain VAE empirically possesses two clear advantages in fMRI signal synthesis: 1) Improved voxel-level reconstruction fidelity and faster convergence with lower validation MSE within fewer epochs; 2) Stronger semantic expressiveness captured in underlying neural patterns indicated by lower validation CLIP loss and higher cross-modal retrieval accuracies.
BrainVAL Pipeline. BrainVAL VAE consists of an encoder-decoder architecture to reconstruct IMRI signals through variational latent distribution. Given an IMRI input VIMRI E >glxn the encoder outputs posterior distribution (( y) parameterized by mean j and log- varianni log d². defining a latent Gaussian distribution ~ M(u, o²). A latent vector is sampled using the reparameterizzaiion trick and passed to the decoder to produce the reconstruction ÒIMRI = D(z). The design helps capture meaningful variations in brain responses while preserving structural consistency.
Training Objectives. To ensure faithful reconstruction and probabilistic, semantically aligned latent space, Brain VAE is optimized with composite objective consisting of three loss terms.

---

1) Reconstruction Loss LMSE: This loss measures the voxel-wise mean squared enror between the reconstructed fMRI signal DIARI and the ground-true -ruth input yIMRI:
MSE [D(z) yンMRIll.
(I)
It enforces fidelity at the voxel level, ensuring that the decoder can reproduce detailed neural activity patterns from the latent representation.
2) KL Divergence Loss LKL: The KL term regularizes the learned posterior g(> (/MRR) to be close to standard normal prior (0,0 I):
LKL (x UIRRI) || (0, 1)).
(2)
This term encourages smoothness in the latent space, ensuring that similar fMRI inputs map to nearby latent codes and that sampling from the prior leads to plausible reconstructions.
3) Contrastive Loss LCLIP: We adopt contrastive learning objective, termed SoftCLIP loss [1] to semantically ground the fMRI latent space and enhance cross-modal alignment. Given an input image y. we extract its visual representation SCLIP Viy) [Ra through a frozen CLIP visual encoder V(-), where m is the number of visual tokens and d is the feature dimension. Concurrently, the [MRI signal 1/MRR is encoded via our brain encoder E-) to produce latent features z & {YMMRI) Rmxdd Alignment is achieved by minimizing:
CCLIP SoftCLIP ZCLIP).
This alignment encourages the latent space to encode semantic information consistent with the visual stimulus, enabling more controllable and meaningful fMRI generation. The final training loss is defined as weighted sum of the above components:
LBaainVAE = LMSE + ŽKLLKL + ACLIPCLLIP.
Here we set ÀKL 0.001 seftly regularize the latent distribution while preserving reconstruction quality, and ACLIP 1000 to ensure fast convergence of semantic alignment within a few epochs.
Through this design, Brain VAE models latent distribution of neural responses conditioned visual stimuli, capturing trial-level variability and consistent semantic patterns. This formulation provides a probabilistic and semantically aligned space for visual-to-fMe synthesis.
2.2 S2N Mapper
Contrastive learning inherently only encourages the alignment of fMRI embeddings with the vector direction of associated visual embeddings, which may lead inconsistent embeddings [44). Hence, we propose the Semantic-to-Neural (S2N) Mapper to improve cross-modal alignment by mapping visual embeddings directly into the fMRI latent space, forming a point-t-ddistribution alignment.
S2N Mapper comprises a typical Transformer (55) module with stacked multi-head self attentttn layers and token- wise feedforward networks. Given visual embedding 2CLIP E R mnd the S2N Mapper implements a non-linear transformation fs2N mm >> I xd to map #CLIP into the fMRI latent space. The ground-truth latent representation measured fMRI signal YEMRA using the pretrained Brain VAE encoder E. Finally, the model is trained by minimizing the voxel-wise mean squared error (MSE) between the aligned latent representation Js2N (ZCLIP) ) and the target IMRI embedding z:
{(ymm))) is obtained by encoding the
AAlign
LsEN MSE(())
(5)
Compared to diffusion- alignment, like Diffusion Prior 44] used in MindEye [47 and Diffusion Transformer 4| used in MindSimulator 4 our one-step mapping strategy eliminates the need for iterative denoising and avoids the training-infereet operating entirely within a semantically structured latent space. Instead of starting from random noise, the S2N Mapper receives well-formed CLIP embeddings input and directly predicts fMRI latent representations that lie within the target manifold. This one-step, point-to-distribution mapping eliminates the need for handcrafted priors, repeated sampling, or post-hoc averaging used in Mind- Simulator 4 facilitating stable and computationally efficient visual-to- to-al-t-fMMRI synthesis process (see Appendix D for details).
distribution gap (see Appendix [) by
5

---

2.3 Visual-to-IMRR Inference
At inference time, our framework generates fMRI signals directly from visual stimuli through a streamlined pipeline. Given an input image æ, we extract its semantic embedding ECLPP V(x) using the frozen CLIP encoder. The S2N Mapper then maps ZCLPP to the aligned latent representation FAlign = (=2LL) which is subsequently decoded by the pretrained VAa decoder D(-) to synthesize the corresponding fMRI response ÚIMRI D(2). This inference process requires no iterative refinement, handcrafted priors, or post-selection procedures, allowing fast and stable visual-to-MMRI synthesis suitable for downstream analysis.
Experimental Settings
We conduct experiments on the Natural Scenes Dataset (NSD) 22. large-scale fMRI dataset in which 8 subjects viewed natural images from the COCO dataset 22 across approximately 40 hours of scanning. Following MindSimulator [4], we focus 4 subjects (Sub-1, Sub-2, Sub-5, Sub-7) who completed all experimental sessions. For each subject, we use 9,000 unique images for training and evaluate on shared set of 1,000 test images, each presented across 3 trials to account for response variability. See Appendix A for additional dataset details.
Implementation Details. We adopt the pretrained OpenCLIP ViT-bigG/14 model [3] as frozen visual encoder to extract semantic embeddings from visual stimuli. Our model is implemented and trained on 4 NVIDIA A100 GPUs (40GB memory per GPU), with training completed within 2 hours. Note that single NVIDIA A100 GPU is sufficient to train a variant of SynBrain (d=1024) with trivial performance degradation (see Appendix Table details).
Training Settings. We explore visual-to-fMRI synthesis across settings and assess whether the generated fMRI signals can effectively support decoding in few-shot scenarios.
i) Subject-specific full-data (40-hour) training: In this setting, we train Brain VAE using the Adam optimizer with (31, (32) = (0.9, 0.999). learning rate of x 10 weight decay of 0.05, and batch size of 24. We apply early stopping to prevent overfitting. The S2N Mapper is optimized with identical hyperparameters and trained for 50,000 steps.
ii) Novel-subject few-shot (1-hour) adaptation: In this setting, we retain the same training protocol as for source subjects but adopt parameter-eeffciem strategy. Specifically, we finetune the entire Brain VAE while updating only the MLP submodules in S2N Mapper's Transformer architecture.
iii) Data augmentation for few-shot fMRI-to-Image decoding: In this setting, we follow the few-shot experimental protocol of MindEye2, with one key modification: we augment the limited real IMRI data using synthetic IMRI signals generated from unseen images, thereby expanding the training set. This setting helps to validate the quality of synthesized fMRI signals.
Evaluation Metrics. We assess model performance across three complementary levels:
i) Voxel-Level Metrics:
To quantify reconstruction fidelity, we compute three voxel-wist similarity metrics between synthesized fMRI signals and all original trials, and report the average scores across trials: mean squared error (MSE), Pearson correlation (Pearson), and cosine similarity (Cosine).
ii) Semantic-Level Metrics: To assess semantic quality, we use MindFyye2 [49] a pretrained
diffusion- IMRIIto-image decoder. We evaluate the semantic alignment between decoded
and original images using four established metrics: Inception Score (Incep) [51). CLIP similarity
(CLIP) 43 EfficientNet distance (Eff) (53), and SWAV distance (SwAV) 77.
iii) Image Retrieval Accuracy: We measure how well fMRI signals retain semantic information by computing cosine similarity between fMRI embeddings and CLIP visual embeddings. We compare two signal sources: (i) raw fMRI (Raw) and (ii) synthetic fMRI signals (Syn) from SynBrain.
Retrieval results are averaged over 30 sampled subsets of 300 candidate images.
Please refer to Appendix B for detailed descriptions of these metrics and other fMRI-to-image decoding evaluation metrics.

---

Visual Stimulos
Rew F-RRI (3-Tria))
Syathesized EMRI
IMRI-to-Inage Visuaiizatinn
Visual Srimulos
Raw FMRR (3-riill)
Synthessaed FMRT
IMRL-in-naae Visualiztion
Figure 4: Visual-to-fMRR synthesis results of SynBrain and fMRI-to-image visualizations.
Voxel-Level
Semantic-Level (via decoding)
Image Retrieval
Method
MSE.
Pearson
Cosine f
Incep
CLIP
Eff L
SWAV
Raw
Syn f
MindSimulator (Trials=1)
403
.366
92.1%
90.4%
.701
.396
MindSimulator (Trials=S)
385
.37
93.1%
91.2%
.689
.391
SynBrain (Trials=1)
139
.687
.739
95.7%
94.3%
.639
.362
84.8%
92.5%
SynBrain (Subl Sub2, lh)
160
619
.675
89.2%
88. 1%
.751
.311
19.3%
67.4%
SynBrain (Subl-Sool 1h)
.244
.704
.765
89.2%
88,
.749
.422
16.8%
54.8% 8%
SynBrain (Subl-SSube 1h)
.151
630
.679
86.8%
84.7%
.783
.453
13.2%
76.5%
Table I: Quantitative visual-to-IMRI synthesis performance comparisons. Top section: Subject- specific performance averaged across 4 subjects, Trials=N denotes sampling repetitions during inference. Bottom section: Few-shot adaptation performance with only I hour of data from the novel subject (Sub2, Sub5, Sub7).
4 Results and Analysis
4.1 Main Results
Subject-specific Visual-to-fMRR Synthesis. We first evaluate the quality of fMRI synthesis under the subject-specific setting, where both training and evaluation are performed on the same individual. As shown in the top section of Table SynBrain (Trials=1) significantly outperforms MindSimu- lator (4) across all voxel-level and semantic-level metrics, despite using only one-shot generation. These results highlight the effectiveness SynBrain producing faithful and semantically consistent neural signals without the need for repeated sampling handcrafted priors used in MindSimulator.
Moreover, SynBrain yields higher image retrieval accuracy from synthesized fMRI signals (92.5%) than from the raw fMRI signals (84.8%). This result suggests that, while raw fMRI are inherently sparse and contain substantial redundancy, SynBrain can distill the task-relevant, semantically aligned components and generate signals that more directly reflect high-level visual content.
Figure 4 illustrates that even repeated trials of the same visual stimulus elicit fMRI responses with noticeable variability in fine details. SynBrain effectively learns to abstract away such fine-grained fluctuations while capturing consistent semantic patterns across trials (see Appendix B for diverse yet semantically consistent synthesis for identical visual stimulus). This is further evidenced by the fMRI-to to-image visualizations via pretrained MindEye2, which reflect high-level concepts aligned with the original visual stimuli (c.g., teddy bear, man, flower, bus) despite fine-grained differences between the synthesized and raw fMRI signals.
Few-shot Subject-adaptive Visual-to-fMRI Synthesis. In this setting, we conduct few-shot adaptation experiments using only hour of data from novel subjects (Sub2, Sub5, Sub7). As shown in the bottom section of Table SynBrain demonstrates robust performance even in this low-resource setting. Notably, the adapted models maintain competitive voxel-level accuracy and high semantic consistency (e.g., CLIP: 84.7%-88.I 18)). despite limited supervision. Interestingly, retrieval from synthetic fMRI signals remains effective (Syn: up to 76.5%), even though raw IMRI-based retrieval performance drops significantly (Raw: 13.2%-19, 19.5%). This semantic gap highlights the value of SynBrain-generated signals in supporting robust vision-all -aiione representations across individuals with minimal adaptation cost.
7

---

LoweLevel
High-Level
Retrieval
Method
PixCorr
SSIM
Alex-2
Alex-5
Incep 1
CLIP
Ett
SwAV
Image
Brain 1
MindEye2 (Ih)
.235
.288
88 88.0%
93.3%
83.6%
80.8%
.798
.459
94,0%
77.6%
MindAligner (1h)
.266
.415
88.2%
93.3%
83.5%
81.8%
800
.459
90.9%
86.9%
(indEE)c((l))-AA((I)))
.243
.419
90.1%
95.1%
85.1%
84.7%
.770
.432
87.9%
82.0%
MindEyc2((hh)DA((2h))
.244
.418
89.9%
95.1%
85.6%
85.0%
.773
.434
84.1%
80.5%
Mind)E)e2(I))+DAAAai))
.246
.422
90.1%%
94.6%
85.8%
84.0%
.772
.433
79.4%
77.7%
Table 2: Comparison of fMRI-to-image decoding performance under few-shot adaptation on Subject 1. MindEye2 and MindAligner are finetuned using only hour of real fMRI data from Subject 1, while our data augmentation (DA) strategy enhances the training set with synthetic fMRI signals generated by SynBrain (Sub2~>Sul from novel images in unseen sessions.
Voxel-Levee
Semantic-LLever (via decoding)
Image Retrieval
Method
MSE |
Pearson
Cosine
Incep t CLIP
Eff J SwAV
Raw
Syn
SynBrain
.079
.715
.769
96.7%
95.9%
.619
.350
94.7%
99.3%
w/o Variation Sampling
.086
.687
.712
88.5%
86.7%
.688
.398
90.2%
88.4%
w/o Contrastive learning
.127
.635
.673
86.2%
84.5%
.694
.426
0.5%
0.4%
w/o S2N Mapper
.105
.564
.661
77.3%
75.0%
.838
.514
94.7%
50.5%
Table 3: Ablation experiments of SynBrain under the subject-specific setting on Subject 1.
Data Augmentation for Few-shot IMRI-to-Image Decoding To improve fMRI-to-image decoding performance in data-limited conditions, we introduce Data Augmentation (DA) strategy that supplements real fMRI recordings with synthetic fMRI signals generated by SynBrain. Specifically, given only hour of real fMRI data from an unseen subject, we synthesize additional fMRI data using novel images from unseen sessions and add them the training set. We evaluate MindEye2 (1h) augmented with varying amounts of synthetic data (1h, 2h, and 4h), and compare it against two baselines: MindEye2 (Ih) and MindAligner (1h), finetuned exclusively on hour of real fMRI data from Subject 1. Here, the Sub2->ubl model of SynBrain is used to synthesize fMRI samples.
reports quantitative results across multiple IMRI-to-image reconstruction and retrieval metrics. Despite using the same 1-hour real dataset, models augmented with synthetic signals consistently outperform the baselines, except for the image retrieval performance. Notably, MfindEye2DDA( Ih) achieves a higher CLIP similarity (84.7% vs. 80.8%), better Inception Score (85.1 vs. 83.6), and stronger brain retrieval performance (82.0% vs. 77.6%). Interestingly, the benefit plateaus or slightly declines with more synthetic data, suggesting that moderate augmentation yields the best balance between diversity and quality. These findings demonstrate that our high-quality synthetic fMRT signals can serve as an effective form of data augmentation, improving iMRI-to-imen to-image decoding in few-shot settings without requiring time-consuming and laborious data collection procedures.
4.2 Ablation Study
To assess the contribution of key components in SynBrain, conduct series of controlled ablation experiments on Subject 1, examining the effects of removing the variational sampling (LKL), contrastive learning (CCrsp), and S2N Mapper. Results summarized in Table 3
Removing this component (i.e., using a deterministic autoencoder) significantly degrades semantic-level alignment (Incep: 96.7% - 88.5% CLIP: 95.9% - 86.7%).
Impact Variation Sampling:
This highlights the importance of distribution-Ievel probabilistic learning for capturing functional consistency behind the inherent variability of neural responses rather than overfitting to specific deterministic patterns.
Impact of Contrastive learning: Removing this component causes semantic retrieval accuracy to collapse (99.3% + 0.5%), indicating that the Brain VAE encoder fails to map neural responses into space. Interestingly, the model still retains non-trivial semantic alignment performance (e.g., CLIP: 84.5%), which suggests that Brain implicitly captures structural and semantic regularities in the fMRI data during reconstruction. The S2N Mapper, trained in the second stage, can leverage this structured latent space to partially align with visual semantics, despite the absence of contrastive guidance.
shared
8

---

Raw DMB1 Reqprraxx
SysBrais EXRI Systhsis
-MR------ass Vissalioiian
Viana Stimali
Subl Suub2
Siieuli
Thna-2
Sabl-Suuhl
Figure 5: Cross-trial and cross-subject brain functional consistency visualization. Left: Comparisons of activation maps between different fMRI trials and our synthesized 1MRI evoked by the same stimuli. Right: Comparisons of activation maps between Sub2 (Full-data, 40h) and Subl-+ubb2 (Few-Shot, 1h) evoked by representative categories of visual stimuli.
The S2N Mapper is introduced after contrastive alignment to project CLIP embeddings directly into the fMRI latent manifold. Eliminating this module leads the most pronounced semantic degradation (CLIP: 95.9% + 75.0%; Inception: 96.7% + 77.3%), highlighting its essential role in bridging the modality gap.
Impact of S2N Mapper:
4.3 Brain Functional Analysis
Neural responses exhibit inherent variability while maintaining consistent functions fundamental property of biological systems that our probabilistic approach explicitly models 14
Analyzing
SynBrain's 's ability to capture this duality offers deeper insight into neural information processing.
Cross-trial Functional Consistency. Neural responses identical stimuli vary across trials due
to physiological noise, spontaneous fluctuations, and intrinsic stochastic properties [36 g 42 Our
analysis of synthesized versus raw fMRI responses (Figure reveals an important principle: category- selective regions (e.g.. fusiform face area) maintain consistent activation patterns across trials despite substantial voxel-level variability [14 40). SynBrain successfully captures this balanced relationship
between variability and consistency, suggesting that trial-to-trial fluctuations follow structured pattern rather than reflecting random noise 12]. The preservation of regional activation amid voxel- level fluctuations supports hierarchical organization of neural variability-wii while low-level sensory details may vary. hige-level semantic representations remain stable. This aligns with theoretical frameworks proposing that neural variability serves as form of probabilistic computation [40], enabling the brain to represent multiple interpretations of input while converging on consistent semantics.
Cross-subject Functional Consistency. Despite individual differences in brain anatomy and activity, similar representational dynamics emerge when processing similar stimuli [45] 655. Our cross-subject adaptation results provide a computational explanation for this phenomenon. When adapting from 1ubject-1 to Subject-2 using only hour of data, SynBrain produces activation patterns closely aligned with those obtained from a model trained on the full Subject-2 dataset, particularly in category- selective regions. This efficient adaptation suggests that individual variability in visual processing highly structured, likely constrained interferes with core semantic representations. The success of our adaptation approach supports the hypothesis that invariant semantic representations can effectively disentangled from subject- specific characteristics [14 40 65] This principle may explain recent findings of preserved neural dynamics across individuals and species [45), suggesting a fundamental organizational mechanism by which the brain balances individual variation with functional consistency.
low-dimensional subspace that minimally
Related Work
Visual-to-IMRI Encoding. Encoding visual input into fMRI responses has been major topic
of interest in neuroscience and computational modeling 37 typically adopt regression-based frameworks to establish mappings from visual features voxel-
18 54). Conventional methods

---

wise neural activity. Some studies aim to improve the quality of input features through task- cisk-specific models (20 [6]] or by optimizing the selection of visual stimuli [31] while others focus on enhancing
the regression function itself through nonlinear architectures 16 61 27
33 S However, most
of these models predict a single deterministic response for each image, which limits their ability to capture the variability inherent in neural data.
More recendly, MindSimulator 4 proposed a generative encoding approach using a diffusion-based model built on a deterministic MLP-based autoencoder. However, its stochasticity is introduced solely at inference time via diffusion sampling started from Gaussian noise distribution, while the generative process still remains deterministic. As such, the system does not learn latent distribution over neural responses, but instead treats random perturbations as input variations, requiring averaged multiple samplings and heuristic post- mest-selection to produce stable and plausible results.
The proposed SynBrain framework addresses these limitations by employing probabilistic learning to model neural variability and introducing a one-step point-to-distriutiion mapping mechanism, allowing stable and semantically aligned fMRI synthesis in a single forward pass.
Few-Shot IMRI-to-Visuu Decoding, The task of fMRI-to-visual decoding aims to reconstruct visual stimuli from recorded IMRI responses, with related neural decoding efforts also explored in the Electroencephalographh (EEG) 19 30 64 59] and Magnetoenceppallgrapp
(MEG) [6 62
domains. Recent approaches typically map fMRI signals into pretrained vision-language embedding spaces (e.g.. CLIP) and decode them into images using powerful generative backbones [8] 52 47
While subject-specific decoding has achieved impressive results, however, collecting high-quality fMRI data is expensive, time-consuming, and subject to strong individual variability- scale subject-specifie training impractical in many scenarios. To address this, recent studies have explored cross-subject few-shot adaptation, where models are adapted to novel individuals using only
large-
I hour of brain recordings 48
46
57
17
In this setting, MindEye2 46 employs ridge regression to align subjects into a shared latent space, followed by a universal decoding module. MindAligner [ further introduces an explicit brain functional alignment model that relaxes the constraint of shared stimuli, achieving state-of oiite-of-theaar decoding performance with minimal learned parameters.
Despite their promise, these methods remain challenged by inter-subject variability under limited supervision. In this work, we investigate whether synthetic fMRI signals can serve as an effective form of data augmentation expanding the number of fMRI-image training pairs and thereby improving visual decoding performance in few-shot cross-subject adaptation settings.
Conclusions
SynBrain advances visual encoding models by explicitly addressing the biological reality of neural variability while preserving functional consistency. Our latent space modeling approach, integrating Brain VAE for cross-tria trial variability modeling and S2N for tiiuuls-to-neural distribution mapping, enables biologically plausible neural response modeling. Experimental results demonstrate superior encoding performance while maintaining neural response characteristics, establishing more accurate digital twin of visual neural circuits.
Broader Impact. SynBrain's probabilistic modeling paradigm extends naturally to other neural modalities and brain regions beyond the visual cortex. This approach has particular potential for brain-computer interface applications, where modeling individual variability while maintaining functional consistency could significantly reduce calibration requirements and improve robustness.
Limitations and Future Direction. Despite these advances, several challenges remain. First, our reliance on pre- traitrained vision models may introduce inherited representational biases that may not perfectly align with neural processing. Second, while we model response variability, we cannot yet ac- count for all variance sources, such as attentional state fluctuations or neuromodulatory effects. These limitations highlight opportunities for further refinement of probabilistic neural encoding approaches. Our findings reveal that individual neural differences occupy structured dimeenniioonll subspaces largely orthogonal to semantic representations, explaining SynBrain's 's efficient cross-subjec transfer capability. This computational principle enables personalized neurotechnology with minimal cali- bration data. Future directions include extending this framework to other sensory modalities and integrating temporal dynamics for multimodal brain modeling, with potential applications spanning fundamental neuroscience research and clinical brain-computer interfaces.
10

---

Acknowledgments
This work was supported by Shanghai Artificial Intelligence I Laboratory and the JC STEM Lab of AT for Science and Engineering, funded by The Hong Kong Jockey Club Charities Trust, the Research Grants Council of Hong Kong (Project No. CUHK 1421322))
References
[1] Hossein Adeli, Sun Minni, and Nikolaus Kriegeskorte. Predicting brain activity using transformers. bioRxiv, pages 2023-05 08, 2023.
[2] Emily J Allen, Ghislain St- Yves, Yihan Wu, Jesse L Brecdlove, Jacob S Prince, Logan T Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, et al. A massive 7t finr datasct to bridge cognitive neuroscience and artificial intelligence. Nature neuroscience, 25(1):116-126, 2022.
[3] Guangyin Bao, Zixuan Gong, Qi Zhang, Jialei Zhou, Wei Fan, Kun Yi. Usman Naseem, Liang Hu, and Duoqian Miao. Wills aligner: A robust multi-subject brain representation learner. arXiv preprint arXiv:2404. 13282, 2024.
[4] Guangyin Bao, Qi Zhang, Zixuan Gong, Zhuojia Wu, and Duogian Miao. Mindsimulator: Exploring brain concept localization via synthetic fmri. arXiv preprint MXXi::503300355: 2025.
[5] Roman Beliy, Navve Wasserman, Amit Zalcher, and Michal Irani. The wisdom of a crowd brains: universal brain encoder. arXiv preprint arXiv:2406. 12179, 2024.
[6] Yohann Benchetrit, Hubert Banville, and Jean-Rémi King. Brain decoding: toward real-time reconstruction of visual perception. arXiv preprint arXiv:2310. 198/2, 2023.
[7] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsu- pervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912 2924,, 2020.
[8] Zijiao Chen, Jiaxin Qing. Tiange Xiang, Wan Lin Yue, and Juan Helen Zhou. Seeing beyond the brain: Conditional diffusion model with sparse masked modeling vision decoding. In JEEE/CVF Conference on Computer Vision and Puttern Recognition, pages 12710-22720) 2023.
[9] Mark M Churchland, Byron Yu, John P Cunningham, Leo Sugrue, Marlene Cohen, Greg Corrado, William T Newsome, Andrew M Clark, Paymon Hosseini, Benjamin B Scott, et al. Stimulus onset quenches neural variability: widespread cortical phenomenon. Nature neuroxcience, 13(3):369-378, 2010.
[10] Bryan Conroy, Ben Singer, James Haxby, and Peter Ramadge. fmri-based inter-subject cortical alignment using functional connectivity. Advances in neural information processing systems, 22, 2009.
[11] Yuqin Dai, Zhouheng Yao, Chunfeng Song, Qihao Zheng, Weijian Mai, Kunyu Peng, Shuai Lu, Wanli Ouyang, Jian Yang, and Jiamin Wu. Mindaligner: Explicit brain functional alignment cross-subject visual decoding from limited fmri data. arXiv preprint arXiv:2502.05033 2025.
[12] A Aldo Faisal, Luc PJ Selen, and Daniel M Wolpert. Noise the nervous system. Nature reviews neuroscience, 9())292--33, 2008.
[13] F Guerreiro Fernandes, M Raemaekers, Freudenburg, and N Ramsey. Considerations for implanting speech brain computer interfaces based on functional magnetic resonance imaging. Journal of Neural Engineeringg 21(3):036005, 2024.
[14] Juan A Gallego, Matthew G Perich, Raeed H Chowdhury, Sara A Solla, and Lee E Miller. Long-term stability of cortical population dynamics underlying consistent behavior. Nature neuroscience, 23(2):260- 270, 2020.
[15] Yuting Gao, Jinfeng Liu, Zihan Xu, Tong Wu, Enwei Zhang, Ke Li, Jie Yang. Wei Liu, and Xing Sun. Softelip: Softer cross-modal alignment makes clip stronger. In Proceedings the AAAI Conference on Artificial Intelligence, volume 38, pages 1860-1868, 2024.
the
[16] Alessandro Gifford, Benjamin Lahner, Sari Saba- Sabay, Martina G Vilas, Alex Lascelles, Aude Oliva, Kendrick Kay, Gemma Roig, and Radoslaw M Cichy. The algonauts project 2023 challenge: How the human brain makes of natural scenes. arXiv preprint arXiv:2301.033 031999998... 2023.
11

---

[17] Zixuan Gong, Qi Zhang. Guangyin Bao, Lei Zhu, Ke Liu, Liang Hu, and Duoqian Miao. Mindtuner: Cross- subject visual decoding with visual fingerprint and semantic correction. arXiv preprint arXiv:2404. 12630. 2024.
[18] Zijin Gu, Keith Jamison, Mert Sabuncu, and Amy Kuceyeski. Personalized visual encoding model construction with small data. Communications Biology, 1382, 2022.
[19] Zhanqiang Guo, Jiamin Wu, Yonghao Song, Jiahui Bu, Weijian Mai, Qihao Zheng, Wanli Ouyang, and Chunfeng Song. Neuro- 3d: Towards 3d visual decoding from egg signals. In Proceedings of the Computer Vision and Pattern Recognition Conference, pages 23870 238800 2025.
[20] Kuan Han, Haiguang Wen, Junxing Shi, Kun-Han Lu, Yizhen Zhang, Di Fu, and Zhongming Liu. Vari- ational autoencoder: An unsupervised model for encoding and decoding fmrr activity in visual cortex. Neurolmage, 198:125- 126-136, 2019.
[21] James V Haxby, J Swaroop Guntupalli, Andrew C Connolly, Yaroslav O Halchenko, Bryan R Conroy. M Ida Gobbini, Michael Hanke, and Peter J Ramadge. A common, high-dimensional model of the representational space in human ventral temporal cortex. Neuron, 72(2):404 416, 2011.
[22] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 16000-16009, 2022.
[23] Tomoyasu Horikawa and Yukiyasu Kamitani. Generic decoding of seen and imagined objects using hicrarchical visual features. Nature communications, 8(1):15037, 2017.
[24] Alexander G Huth, Wendy A De Heer, Thomas L Griffiths, Frédéric E Theunissen, and Jack L Gallant. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature, 532(7600) 2016.
[25] Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolutional neural networks. Advances in neural information processing systems, 25, 2012.
[26] Ruixing Liang. Xiangyu Zhang, Qiong Li, Lai Wei, Hexin Liu, Avisha Kumar, Kelley M Kempski Leadingham, Joshua Punnoose, Leibny Paola Garcia, and Amir Manbachi. Unidirectional brain- computer interface: Artificial neural network encoding natural images to fmri response the visual cortex. In International Conference on Acoustics, Speech and Signal Processing, pages 1851-1855. IEFF, 2024.
[27] Ruixing Liang, Xiangyu Zhang, Qiong Li, Lai Wei, Hexin Liu, Avisha Kumar, Kelley M Kempski Leadingham, Joshua Punnoose, Leibny Paola Garcia, and Amir Manbachi. Unidirectional brain- comptco interface: Artificial neural network encoding natural images to fmri response in the visual cortex. In ICASSP 2024-2024 JEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 1851- IEEE, 2024.
[28] Sikun Lin, Thomas Sprague, and Ambuj K Singh. Mind reader: Reconstructing complex images from brain activities. Advances in Neural Information Processing Systems, 35:29624-29636. 2022.
[29] Tsung-Yr Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European Conference on Computer Vision, pages 740-755, 2014.
[30] Weiheng Lu, Chunfeng Song, Jiamin Wu, Pengyu Zhu, Yuchen Zhou, Weijian Mai, Qihao Zheng, and Wanli Ouyang. Unimind: Unleashing the power of Ilms for unified multi-task brain decoding. arXiv preprint arXiv:2506. 18962, 2025.
[31] Andrew Luo, Maggie Henderson, Leila Wehbe, and Michael Tarn. Brain diffusion for visual exploration: Cortical discovery using large scale generative models. Advances in Neural Information Processing Systems, 36, 2024.
[32] Andrew F Luo, Jacob Yeung, Rushikesh Zawar, Shaurya Dewan, Margaret M Henderson, Leila Wehhe, and Michael Tarr. Brain mapping with dense features: Grounding cortical semantic selectivity in natural images with vision transformers. arXiv preprint 2xXi:2411.05566 2024.
[33] Shuxiao Ma, Linyuan Wang, Senbao Hou, Chi Zhang, and Bin Yan. Large-scale parameters framework with large convolutional kernel for encoding visual fmri activity information. Cerebral Cortex. 34(7), 2024.
[34] Weijian Mai, Jian Zhang, Pengfei Fang, and Zhijun Zhang Brain condiiionii multimodal synthesis: A survey and taxonomy. IEEE Transactions Artificial Intelligence, 2024.
12

---

[35] Weijian Mai and Zhijun Zhang. Unibrain: Unity image reconstruction and captioning all in one diffusion model from human brain activity. arXiv preprint arXiv:2308.07423, 2023.
[36] Eve Marder and Mare Goaillard. Variability, compensation and homeostasis in neuron and network function. Nature Reviews Neuroscience, 7(7):563-574, 2006.
[37] Tom M Mitchell, Svetlana V Shinkareva, Andrew Carlson, Kai-Min Chang, Vicente L Malave, Robert A Mason, and Marcel Adam Just. Predicting human brain activity associated with the meanings of nouns. science, 320(5880): 191-1195, 2008.
[38] Thomas Nasclaris, Kendrick N Kay, Shinji Nishimoto, and Jack L Gallant. Encoding and dccoding in fmri. Neuroimage, 56(2)400-410, 2011.
[39] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza. Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint WxXiv:2304.07133, 2023.
[40] Gergö Orbán, Pietro Berkes, József Fiser, and Máté Lengyel. Neural variability and sampling-bast based probabilistic representations the visual cortex. Newron, 92(2):530 543, 2016.
[41] William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 4195-4205, 2023.
[42] Adrián Ponce-Alvarez, Alexander Thiele, Thomas D Alhright, Gene R Stoner, and Gustavo Deco. Stimulus- dependent variability and noise correlations in cortical mt neurons. Proceedings of the National Academy of Sciences, 110032):13162-1311,,, 2013.
[43] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In ICML, pages 8748 8763. PMLR, 2021.
[44] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06122 1(2):, 2022.
[45] Mostafa Safaie, Joanna C Chang, Junchol Park, Lee E Miller, Joshua T Dudman, Matthew G Perich, and Juan A Gallego. Preserved neural dynamics across animals performing similar behaviour. Nature, 623(7988):765-7777 2023.
[46] P. S. Scotti, M. Tripathy, C. K. T. Villanueva, R. Kneeland, T. Chen, A. Narang, C. Santhirasegaran, Xu, T. Naselaris, and K. A. et al. Norman. Mindeye2: Shared-subject models enable fimri-to-image with hour of data. In ICML, 2024.
[47] Paul Scotti, Atmadeep Banerjee, Jimmic Goode, Stepan Shabalin, Alex Nguyen, Aidan Dempster, Nathalic Verlinde, Elad Yundler, David Weisberg, Kenneth Norman, et al. Reconstructing the mind's eye: finri-to image with contrastive learning and diffusion priors. Advances in Neural Information Processing Systems, 36, 2024.
[48] Paul Scotti, Atmadeep Banerjee, Jimmic Goode, Stepan Shabalin, Alex Nguyen, Aidan Dempster, Nathalic Verlinde, Elad Yundler, David Weisberg, Kenneth Norman, et al. Reconstructing the mind's eye: firi-to- image with contrastive learning and diffusion priors. Advances in Neural Information Prucessing Syxtems, 36, 2024.
[49] Paul Scotti, Mihir Tripathy, Cesar Kadir Torrico Villanueva, Reese Kneeland, Tong Chen, Ashutosh Narang. Charan Santhirasegaran, Jonathan Xu, Thomas Naselaris, Kenneth A Norman, et al. Mindeye2: Shared-subject models enable fmri-to-image with hour of data. arXiv preprint arXiv:2403. /207, 2024.
[50] Ghislain St- Yves, Emily Allen, Yihan Wu, Kendrick Kay, and Thomas Naselaris. Grain-optimize deep neural network models of human visual areas learn non-hierarchical representations. Nature communica- tions, 14(1);3329, 2023.
[51] Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethinking the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 2818-2826, 2016.
[52] Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. pages 14453-14465 14463, 2023.
[53] Mingxing Tan and Quoc Le. Efficientnet: Rethinking model scaling for convolutional neural networks. In International Conference on Machine Learning, pages 6105-611 6014 2019.
13

---

[54] Jenry Tang, Meng Du, Vy Vo, Vasudev Lal, and Alexander Huth. Brain encoding models based on multimodal transformers transfer across language and vision. Advances in Neural Information Processing Systems, 36, 2023.
[55] Ashish Vaswani, Noam Shazeer, and Niki Parmar. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
[56] Aria Y Wang, Kendrick Kay, Thomas Naselaris, Michael J Tam, and Leila Wehbe. Better models of human high-level visual cortex emerge from natural language supervision with large and diverse dataset. Nature Machine Intelligence, 5(12):1415 1426, 2023.
[57] S. Wang, Liu, Tan, and X. Wang, Mindbridge: cross-subject brain decoding framework. pages 11333-11344 11342, 2024.
[58] Zhou Wang, Alan Bovik, Hamid R Sheikh, and Eero Simoncelli. Image quality assessment: error visibility to structural similarity. IEEE transactions on image processing, 13(4))600- 612, 2004.
[59] Jiamin Wu, Zichen Ren, Junyu Wang, Pengyu Zhu, Yonghao Song, Mianxin Liu, Qihao Zheng, Bai, Wanli Ouyang, and Chunfeng Song. Adabrain-bench: Benchmarking brain foundation models for craputcomputer interface applications. arXiv preprint arXiv:2507.098: 2025.
[60] Weihao Xia, Raoul de Charette, Cengiz Öztireli, and Jing-Hao Xue. Umbrae: Unified multimodal brain decoding. In Eurupean Conference on Computer Vision, 2024.
[61] Huzheng Yang, James Gee, and Jianbo Shi. Memory encoding model. arXiv preprint a7X:1:2308.01175 2023.
[62] Yiqian Yang. Hyejeong Jo, Yiqun Duan, Qiang Zhang, Jinni Zhou, Won Hee Lee, Renjing Xu, and Hui Xiong. Mad: Multi-alignment meg-to-text decoding. arXiv preprint arXiv:2406.01512 2024.
[63] Mariam Zabihi, Seyed Mostafa Kia, Thomas Wolfers, Stijn de Boer, Charlotte Fraza, Richard Dinga, Alberto Llera Arenas, Danilo Bzdok. Christian F Beckmann, and Andre Marquand. Nonlinear latent repre- sentations of digh-diimnsiooal task-fmri data: Unveiling cognitive and behavioral insights in heterogeneous spatial maps. Plus one, 193383333 2024.
[64] Yuchen Zhou, Jiamin Wu, Zichen Ren, Zhouheng Yao, Weiheng Lu, Kunyu Peng. Qihao Zheng, Chunfeng Song. Wanli Ouyang, and Chao Gou. Csbrain: cross-scale spatiotemporal brain foundation model for eeg decoding. arXiv preprint arXiv:2506.23075. 2025.
[65] Zhu, Chunfeng Song, Wanli Ouyang, Shan Yu. and Tiejun Huang. Neural representational consistency emerges from probabilistic neural-bchavione representation alignment. arXiv preprint 2rx1v:2505.0433/,, 2025.
[66] Vadim Zotev, Jessica R McQuaid, Cidney R Robertson -entta, Anne K Hittson, Tracey V Wick, Josef M Ling, Harm J van der Horn, and Andrew R Mayer. Validation of real-time fmri neurofeedback procedure for cognitive training using counterbalanced active-sham study design. Neurolmage, 290: 120575, 2024.
14

---

NeurIPS Paper Checklist
The checklist is designed to encourage best practices for responsible machine learning research, addressing issues of reproducibility, transparency, research ethics, and socictal impact. Do not remove the checklist: The papers not including the checklist will be desk rejected. The checklist should follow the references and follow the (optional) supplemental material. The checklist does NOT count towards the page limit.
Please read the checklist guidelines carefully for information on how to answer these questions. For each question in the checklist:
You should answer [Yes], , [No] or [NA].
[NA] means either that the question Not Applicable for that particular paper or the relevant information is Not Available.
Please provide short (1-2 sentence) justification right after your answer (even for NA).
The checklist answers are an integral part of your paper submission. They are visible to the reviewers, area chairs, senior area chairs, and ethics reviewers. You will be asked to also include (after eventual revisions) with the final version of your paper, and its final version will published with the paper.
The reviewers of your paper will be asked use the checklist as one of the factors their evaluation. While "[Yes] is generally preferable to "[No]" ", it is perfectly acceptable to answer "[No] pprovided proper justificaiion is given (e.g., "error bars are not reported because would be too computationally expensive" or "we were unable to find the license for the dataset we used"). In general, answering binary way, we
"[No] or "[NA]" is not grounds for rejection. While the questions are phrased acknowledge that the true answer is often more nuanced, so please just use your best judgment and write justification to elaborate. All supporting evidence can appear either in the main paper or supplemental material, provided in appendix. If you answer [Yes] to a question, in the justification please point to the section(s) where related material for the question can found.
IMPORTANT, please:
Delete this instruction block, but keep the section heading "NeurIPS Paper Checklist",
Keep the checklist subsection headings, questions/answers and guidelines below.
Do not modify the questions and only use the provided macros for your answers.
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Main claims have been made in the abstract and introduction to reflect our contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the
contributions made in the paper and important assumptions and limitations. A No
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
15

---

Justification:
The limitation of our work has been discussed in the Conclusion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approxinations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g. if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proof'
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: The paper mainly provides experimental results and does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Implementation details are provided in the Experimental Setting section.
Guidelines:
16

---

The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open- source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors welcome deseribe the particular way they provide for reproducibilityy In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification:
Code submitted as additional supplementary material.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.. cc/
otteeeeuubbisiioooooooooo.o for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https ctipps.cc/publi///uue
Codeeuumiiii wettsbbmmssiiioi ssionoolicy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
17

---

Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental setting/details
Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Question:
Answer: [Yes]
Justification:
Implementation details are provided in the Experimental Setting section.
Guidelines:
The answer NA means that the paper does include experiments.
The experimental setting should be presented in the core the paper to a level detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: We have made comprehensive qualitative and quantitative comparisons be- tween our approach and state-of-the-art methods in this field.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split. initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еттог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We have provided information about the computer resources of our experiments in the Experimental Setting section.
Guidelines:
The answer NA means that the paper does not include experiments.
18

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer: [Yes]
Justification: This work conforms with the NeurIPS Code of Ethics in every respect.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: Broader impacts have been discussed in the Conclusion section.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper poses such risks.
19

---

Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best faith effort.
Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification:
We have respected and cited all original assets.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperssithcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re- re-packaged, both the original license and the license of the derived asset (if has changed) should provided.
If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper does not release new assets.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
20

---

Answer:
[NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Question:
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: We only use LLMs for formatting purposes.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non non-standard components.
Please refer to LLM policy futtps://neuripg: ct/ http://sc..ccoooeeeee/////////// for what should or should not be described.
21

---

A NSD Dataset
In this study, we leverage the largest publicly available fMRI-image dataset, the Natural Scenes Dataset (NSD), which encompasses extensive 7T fMRI data collected from eight subjects while they viewed images from the COCO dataset. Each subject viewed each image for 3 seconds and indicated whether they had previously seen the image during the experiment. Our analysis focuses on data from four subjects (Sub-1, Sub-2. Sub-5, and Sub-7) who completed all viewing trials. The training dataset consists of 9,000 images and 27,000 IMRI trials, while the test dataset includes 1,000 images and 3,000 fMRI trials, with up repetitions per image. It is important note that the test images are consistent across all subjects, whereas distinct training images are utilized.
We used preprocessed scans from NSD for functional data, with resolution of 1.8 mm.. Our analysis involved employing single-trial beta weights derived from generalized linear models, along with region-of-interest (ROI) data specifie to early and higher (ventral) visual regions as provided by NSD. The ROI voxel counts for the respective four subjects are as follows: [15724, 14278, 13039, 12682]. Detailed fMRI preprocessing procedures and additional information be found in the source paper (2) and on the NSD website
B Evaluation Metric
B.1 MRRIto-Imagge Decoding Model.
i) Subject-specific setting: All semantic-level metrics are obtained by decoding the synthesized (MRI using a fixed MindEye2 model trained on the corresponding subject using 40 hours of data.
ii) Few-shot adaptation setting: All semantic-level metrics are obtained by decoding the synthesized fMRI using a fixed MindEye2 model trained on the corresponding subject using I hour of data.
B.2 Visual-to-IMRI Synthesis Metries
We provide detailed descriptions of the evaluation metries used to assess SynBrain's 's performance in visual-to-fMRI synthesis across voxel-level accuracy, semantic-level alignment, and cross- -odall retrieval capability.
Voxel-Level Metrics. To evaluate the fidelity of synthesized fMRI signals, we compare the gen- erated output with ground-truth fMRI recordings for each stimulus on a voxel-by-voxee basis. The following three metrics are computed and averaged across all test. fMRI trials: i) MSE: Computes the mean squared error between synthetic and ground-truth fMRI signals; ii) Pearson: Measures the linear Pearson correlation coefficient between synthetic and ground -tuth fMRI signals; iii) Cosine: Evaluates the angle-based cosine similarity between synthetic and ground-truth fMRI signals.
Semantic-Level Metries via IMRI-to-Imann Image Decoding. To assess whether the synthesized fMRI signals preserve the underlying semantic content of the input stimulus, we leverage the pretrained fMRI-to-image decoder MindEye2 [49] to transform synthetic fMRI into images. The decoded images are then compared with the original ground-truth images using multiple semantic metrics: i) Incep: two-way comparison of the last pooling layer of Inception V3; ii) CLIP: A two-way comparison of the output layer of the Imae-Image model; iii) Eff: A distance metric gathered from EfficientNet-B 1 model; iv) SwAV: A distance metric gathered from SwAV-ResNet50 model.
A two-way comparison evaluates the accuracy percentage by determining whether the original image embedding aligns more closely with its corresponding brain embedding or with randomly selected brain embedding.
Image Retrieval Accuracy. We evaluate cross-modal retrieval performance between CLIP visual embeddings and fMRI embeddings from two sources: i) Raw: The raw fMRI signal from the test set; ii) Syn: The synthetie fMRI signal of SynBrain from the corresponding visual input. We compute the cosine similarity between fMRI embeddings encoded by Brain VAE and 300 candidate image embeddings extracted from test images, with one being the ground-truth visual stimulus for the IMRI
hhnnes//nenuraassessaaaattttoogt
22

---

Algorithm Brain VAE Architecture
1: Input: IMRI signal æ, CLIP visual representation ECLIP
2: Encoder:
3: h Encoder(
Hierarchical Conv+Attn backbone
4: Pre-projector (MLP) for Mean LogVar:
5: h = LayerNorm /)) + GELU
6: h Linear (4096 d) + LayerNorm GELU
d 2048
7: h Linear(d + d) - LayerNorm - GELU
8: μ or log o² Linear(d + 1664)
9: Sampling:
NH,, x²)
11: Post-Projector (MLP):
2 LayerNorm(2 (z) GELU
13:
Linear(1664 - d) + LayerNorm + GELU
D d = 2048
14: 2 = Linear(d + LayerNorm + GELU
15: h' = Linear(d 4096)
16: Decoder:
17:
f = Decoder V.)
V. denotes the voxel count for subject S
18: Loss:
19:
C MSELLss 8) KLWeight KL(H, + CLIPWeight CLIPLoss 2CUP))
20: return
data. Retrieval performance is evaluated by calculating the average Top-I retrieval accuracy (with chance level 1/300) and repeating the process 30 times to account for batch sampling variability.
B.3 Few-shot IMRI-to-Image Decoding Metries
We follow MindEye2 49 evaluate few-shot IMRI-to-image decoding performance in low-level, high-level, and train-image cross- moosl retrieval.
Low-Level Metries. These metrics assess fundamental perceptual image features, providing in- sights into the visual content and structure of the image. measured by: i) PixCorr [10]: Pixel-level correlation between reconstructed and test images; ii) SSIM [58]: Structural similarity index; iii) AlexNet 255: Alex-2 and Alex-5 are the 2-way comparisons of the second (early) and fifth (middle)
layers of AlexNet, respectively.
High-Level Metrics.
Same as Semantic-Level Metrics via fiRItto-te Decoding Section
B.2
Retrieval Metrics. For image retrieval, we compute the cosine similarity between the predicted fMRI embedding and 300 candidate image embeddings extracted from test images, with one being the ground-truth visual stimulus for the fMRI data. Retrieval performance is evaluated by calculating the average Top-1 retrieval accuracy (with a chance level of 1/300) and repeating the process 30 times to account for batch sampling variability. For brain retrieval, we follow the same procedure, but with brain and image samples interchanged.
C Network Architecture
C.1 Brain VAE Architecture
We provide detailed specifications of the BrainVAE architecture, including the encoder, projector modules, and decoder.
Overall Architecture. As shown in Algorithm|] I Brain VAE consists of an encoder, two projection modules (for mean and log-variance), a stochastic sampling step, a post-projector, and decoder. The input fMRI signal E RIXV. is first encoded into a hidden feature h E 2R256x through multi-resolution convolutional encoder. The mean y and log-variance log o² of the latent Gaussian distribution obtained by passing through two identical MLP-based projerrjcssos. Each projector
23

---

Algorithm 2 Brain Encoder Architecture
1: Input: fMRI signal x E RIxV,
V. denotes the voxel count for subject S
2: I < ConviD(x, out, channels = 128, kernel = 1, stride = padding = 0)
I f AdaptiveMaxxPool1((a output 8192)
4: H + [x]
initialize hidden state list num_blocks-33 ch_mult=[1, 4]
5: for 1 to num _lockss do 128 x ch ch_mult[i-i
6:
7: Cout 128 ch [n_mult//]
8:
for = 1 to num_ro moss_bsocssses do
num res blocks =
9:
+ ResnetBlock(H[-I] Gn. Caut)
10:
Append h to
end for
12:
i num_doww__blee blocks then
13:
+ D-DDonsample
14:
Append h to H end
15:
16: end for
17: h - H[-1]
18: h - (esmetBlock(h)) - SelfAttention(h) ResnetBlock(h)
Middle block
19: h + LayerNorm(h) + SiLU(h)
20: h ConviD(h, out_ channels 256, kernel 1, stride 1, padding 0)
21: return E IR2 256x 1096
begins with LayerNorm and GELU activation, followed by two hidden Linear layers with configurable dimension (default d 2048), and ends with final projection 2556x 1664
latent representation ~ (i, o²) is then sampled and passed through post-projector of identical TR. 256x 1661 back to h 2556x1096 The decoder takes h and reconstructs the fMRI signal E R V,, where V, denotes the subject-specific voxel count for subject S.
structure, which maps
The training objective combines three components: (1) an l reconstruction loss between and â. (2) KL divergence between the posterior and prior distributions, and (3) contrastive loss aligning the latent with the CLIP embedding ZCLIP of the visual stimulus.
Encoder Architecture. The encoder design is detailed in Algorithm 2 The input fMRI signal E RRXV, first passes through ID convolution layer with 128 output channels and kernel size of 1, followed by adaptive max pooling to fixed length of 8192. This allows the model to process inputs from different subjects without introducing subject-specifie parameters.
The core of the encoder is hierarchical ResNet-style backbone with three resolution levels, specified by mb_m [1, 2, 4, 4]. Each level includes two residual blocks, and downsampling is applied only after the first level (i.e., num_down, htt__ddwmbblcccke blocks = 1). A middle block with a self-attention layer is inserted between additional residual blocks. After the final ResBlock, the feature normalized, activated using SiLU, and projected via a ID convolution to produce the output / E IR 256 4096 4096
Decoder Architecture. The decoder mirrors the encoder in symmetric fashion. takes the post-projected feature h/' E 256x 1096 as input and processes it through a hierarchical ResNet-style upsampling backbone. The initial convolutional layer increases the feature dimensionality to match the resolution of the encoder output. Then, residual blocks and upsampling layers applied across three resolution levels (reversing the encoder's channel multipliers ch_mult = [4, 4, 2, 1)). One upsampling operation is performed at the final resolution stage (i.e., num_up_br blocks 1), consistent with the encoder's downsampling configuration.
A middle block, identical in structure to the encoder's s. includes a self-attention layer and two ResNet blocks. After final upsampling stage, the decoder produces an intermediate sequence of length 8192. Since the number of voxels V,, may differ across subjects, we apply 1D linear interpolation step to resample the sequence to exactly match the target length V,, final 1D convolution with output channels 1 maps the feature to the reconstructed IMRI signal è e I IXV.

---

Ablation Study on Latent Dimensionality. To investigate the effect of latent dimensionality on synthesis performance. we conduct an ablation study by varying the number of downsampling blocks in Brain VAE from 1 to 3. This progressively reduces the latent dimension of the fMRI representation from 4096 to 2048 and 1024, respectively. As shown in Table 4 using only one down block (i.e., d=4096) consistently achieves the best performance across voxel-level, semantic-level and retrieval metrics. In contrast, deeper compression leads to a gradual decline in synthesis quality. These results suggest that preserving higher-dimeenionall latent features helps retain fine-grained temporal structure in fMRI signals, which is critical for semantic fidelity and voxel-level reproduction.
Voxel-Level
Semantic-Level (via decoding)
Retrieval
Blocks
MSE |
Pearson f
Cosine
Incep t CLIP f
Eff | SwAV +
Raw f
Syn f
(d=4096)*
.079
.715
.769
96.7%
95.9%
.619
.350
94.7% %
99.3%
2 (d-2048)
.022
.703
.746
95.6%
95.0%
.622
.356
94.1%
98.9%
3 (d=1024)
.087
.701
.733
95.2%
94.5%
.643
.363
93.5%
98.0%
Table 4: Quantitative Assual-o-fMRI synthesis performance with different numbers of down blocks in BrainVAE under subject-specific setting on Subject 1. * means the final setting adopted in this work.
C.2 S2N Mapper Architecture
The S2N (Stimulustto-Nuural) Mapper is implemented as a Transformer encoder that maps CLIP visual embeddings fMRI latent representations. We use 8-layer Transformer with 13 attention heads per layer. The input to the mapper is the CLIP visual representation 2CLIP E IR 256 1664 where 256 is the number of tokens and 1664 is the token embedding dimension derived from the OpenCLIP bigG model. To retain positional information, we use fixed sinusoidal positional encodings, which added the input before the first self-attention layer. Each Transformer block follows a pre- layer normalization design and includes multi-head self-attention, a feed-forward MLP with GELU activation, and residual connections.
The S2N Mapper performs a point-to-diste co-distribution transformation by mapping visual embeddings to the center of iMRI latent distributions 2PRRI 256x 1664, which can be used directly or after sampling to synthesize realistic and semantically aligned fMRI responses.
Ablation Study on Number of Attention Layers. We investigate how the number of Transformer layers in the S2N Mapper affects visual-to-fMRR synthesis quality. As shown in Table we compare 4. 8, and 12-layer configurations under the subject-specific setting on Subject 1. The 8-layer model achieves the best overall performance. while increasing the number of layers 12 does not yield further improvements. Hence, we adopt the 8-layer configuration as the final setting in this work.
Voxel-Level
Semantic-Level (via decoding)
Retrieval
Layer
MSE |
Pearson t Cosine
Incep t
CLIP t
Eff
SwAV
Raw t Syn
4
.086
.704
.750
95.8%
95.2%
.632
.358
94.7%
98.5%
8*
.079
.715
.769
96.7%
95.9%
.619
.350
94.7%
99.3%
12
.080
.715
.768
96.6%
96.0%
.620
.350
94.7%
99.3%
Table 5: Quantitative visual-to-IMRI synthesis performance with different numbers of attention layers in S2N Mapper under the subject-specific setting on Subject indicates the final setting adopted in this work.
D Comparison between Diffusion Transformer and S2N Mapper
Distribution Mismatch in Diffusion Transformer. MindSimulator leverages the Diffusion Transformer (DiT) [1 to generate fMRI representations conditioned on visual prompts through iterative denoising. However, as shown in Figure we observe a significant distributional mismatch between training and inference phases in this setup.
25

---

IMRI Representations vs. Noisy IMRI Representations
Pure Gaussian Noise (Test)
IMRI-to-Image
Noisy IMRI Representations (Train)
Visualizations
Avg. Euclidean 0.4663
Avg.Euclidean Distance
SynBrain
Brain VAE-DiT
10-
10-
2-
29
-24
Figure 6: UMAP visualizations of distribution gap in DiT and fMRI-to MRI-to-imaae visualizations. Left: Noisy fMRI representations (blue) used for DiT training still lie close to the original fMRI representa- tions (orange), but far away from pure Gaussian noise (green) used for DiT inference. showing clear distribution gap between training and testing stages in DIT. Right: SynBrain (Brain VAE-S2N, one- step mapping started from original fMRI representations) produces consistent images compared to Brain VAE-DiT (multi-step denoising started from Gaussian noise).
realistic and semantically
Voxel-Level
Semantic- (via decoding)
Retrieval
Method
MSE 4 Pearson f Cosine
Incep f CLIP f
Eff
SwAV +
Raw
Syn f
Brain VAE-DIT
.088
.689
.748
94.8%
93.5%
.642
.376
94.7%
93.2%
SynBrain .079
.715
.769
96.7%
95.9%
.619
.350
94.7%
99.3%
Table 6: Quantitative comparison of fMRI synthesis performance between SynBrain (Brain VAE-S2N) and BrainVAE-Dii under a subject-specifie setting Subject 1.
i) Training:
During training. DiT learns to denoise noisy fMRI latent representations formed by adding Gaussian noise to ground-truth fMRT signals. As visualized in Figure Left, these noisy representations (blue) still lie close to the original fMRI manifold (orange), maintaining structural similarity (Avg. Euclidean Distance 0.4663).
ii) Inference: At test time, however, the model must denoise pure Gaussian noise (green), which is far from the 1MRI manifold (blue; Figurel-Midde The resulting distribution gap (Avg. Euclidean Distance 3.5651) leads to unstable generation quality and typically requires multiple samples and heuristic post- selectenec
S2N Mapper for Direct Semantic Alignment. To address this distribution gap, we propose replacing the diffusion transformer with one-step Semantic-to-Nentt The resulting model, SynBrain (Brain VAE-S2N,, directly maps to the fMRI latent distribution from CLIP embeddings without iterative sampling. This approach ensures distributional consistency and semantic grounding throughout the generation process.
(S2N) Transformer Mapper.
Performance Comparison. Table compares Brain VAE-DiT with SynBrain VAE-S/V in the task of visual-to-IMRI synthesis. Both models share the same VAE-based latent architecture, but differ in how visual representations are mapped to fMRI latent distributions.
SynBrain consistently outperforms Brain VAE-DiT across all evaluation levels. It yields higher semantic fidelity as measured by Incep (96.7% 7% vs. 94.8%) and CLIP similarity (95.9% vs. 93.5%), and significantly improves retrieval accuracy of synthetic fMRI signals (Syn: 99.3% vs. 93.2%). Tigure(6-Righ presents qualitative examples from both models. SynBrain produces more realistic and semantically coherent images than Brain VAE-DiT, which occasionally exhibits semantic drift, likely due to initialization from unstructured Gaussian noise. These results demonstrate that SynBrain more effectively preserves the semantic content of visual inputs in the synthesized fMRI representations. By
26

---

Visual Stimulus
Raw f-MRI (3-Trial)
Synthesized E-MRI (3-Sample, nf
IMRI-to- -mmag Visualization
Sample-1
Sumple-2
Sample-3
ااأاارة dal
Figure Semantically consistent synthesis of SynBrain under stochastic sampling with nf=l.
eliminating the distribution mismatch introduced by denoising from noise, the S2N mapper provides a more stable, interpretable, and semantically grounded solution for visual-to-fMM synthesis.
Semantically Consistent Synthesis under Stochastic Sampling
To evaluate the generative robustness of SynBrain, we investigate its ability to synthesize diverse yet semantically consistent fMRI responses by sampling from the latent distribution. Given distributional center Allggn predicted by the S2N Mapper, we introduce stochastic variation by adding Gaussian noise scaled by a noise factor nf. such that:
Allggn + nf E, N(0,11
Figure presents the synthesis results for a strong noise setting, nf SynBrain can produce fMRI responses that yield highly consistent visual outputs across multiple samples. This suggests that the learned latent space is semantically structured and resilient stochastie perturbations. Empirically, reducing the noise factor (nf < 1) leads to more stable fMRI synthesis with reduced variability. When nf = 0, sampling disabled and SynBrain directly uses the predicted distributional center ZAlign. which may produce single but more realistic and faithful response.
27

---

Visual
VoxeI-Level
Semantic-Lovel (via decoding)
Image Retrieval
Method
Encoder
MSF.).
Pearson f
Cosine 1
Ineep
CLIP
Eff
SwAV
Raw
Syn f
MinuSimulator
CLIP
417
326
92.8%
89.8%
.714
.402
CLIP+
.079
.715
.769
96.7%
95.9%
.619
.350
94.7%
99.3%
SynBrain
MAE
097
.676
.737
91.0%
90.5%
.705
.399
68.6%
83. 7%
DINOv2
089
.679
.728
92.6%
91.8%
.701
.375
76.3%
87.6%
Table 7: Visual-to-IMRR synthesis performance with various visual encoders.
These results demonstrate that SynBrain supports structured one-to one-to-many risual-tofMMR generation and allows controllable sampling of plausible neural responses under varying uncertainty conditions.
F Vision Model Comparison
In this section, evaluate the impact of visual encoder choice by replacing CLIP in SynBrain with two advanced visual foundation models, DINOv2 [39 and MAE [22]. As shown the table below, CLIP achieves the best overall performance across voxel-level and semantic levels. We attribute this to its vision-language contrastive objective, which provides stronger semantic supervision compared purely visual models like DINOv2 and MAE. Nonetheless, the relatively strong performance of these alternatives indicates that SynBrain remains robust across different encoder choices.
We also include comparison with MindSimulator (5-Trial), which likewise uses CLIP its visual encoder. Notably, MindSimulator averages results over five generated fMRI trials to improve performance, whereas SynBrain achieves superior results with just single trial. This suggests that SynBrain benefits not only from a strong encoder but also from a more effective model design and training strategy.
G Additional Baselines
Subject-specific Baselines. Here conduct additional comparisons with representative baselines: (i) GNet [50), and (ii) Linear Regression (LinearReg), a classical deterministic voxel- wise visual encoding model, using simple two-layer linear architecture with 4.006-dimensionall hidden space. As shown in Table [8, SynBrain achieves consistently superior results across multiple levels, highlighting its effectiveness in fMRT synthesis with both voxel-level structure and semantic consistency.
Subject-adaptive baselines. Taking MindSimulator 4 as a cross-subject adaptation baseline is limited by two practical constraints. MindSimulator is not open-sourced and incorporates several hand-crafted components (e.g., resting-state initialization, custom noise injection). making it difficult to reproduce; ii) MindSimulator's Autoencoder is subject-specific and tightly coupled to fixed voxel dimensions, rendering it incompatible with cross-subject scenarios where ROI sizes vary. These limitations prevent a direct and meaningful comparison between SynBrain and MindSimulator in the context of cross- cross-subject subject adaptation.
To establish a transparent and reliable baseline, we implemented Linear Regression (LinearReg), the most classical encoding model, using simple two-layer linear architecture. The first layer maps CLIP features a dinensnnsii hidden space, and the second layer projects subject-specific voxel responses. During few-shot adaptation, we freeze the first layer and train only the second layer using 1-hour data from the target subject. As shown Table and Table 9 SynBrain consistently outperforms Linear Regression across voxel-wise correlation, semantic alignment, and image retrieval accuracy, demonstrating strong generalization and cross-subject transferability under data-limited scenarios.
28

---

VoxeI-Level
Semantic-Level (via decoding)
Image Retrieval
Method
MSE
Pearson
Cosine
Incept 1
CLIP f
Eff'
SwAV +
Raw t
Syn
LincarReg
.102
.666
.693
85.8%
83.7%
.759
.454
GNet
.092
.707
.740
87. 87.7%
85.0%
.730
.428
SynBrain
.079
.715
.769
96.7%
95.9%
.619
.350
94.7%
99.3%
Table 8: Subject-specific visual-to-IMRR synthesis baseline performance on Subject
Voxel-Level
Semantic-Level (via decoding)
Image Retricval
Method (LinearReg)
MSE /
Pearson f
Cosine
Incep f
CLIP f
Eff 4
SwAV 4
Raw f
Syn f
Subl Sub2 (1h)
.288
.822
.550
76.6%
76.7%
.861
.514
Subl Sub5 (1h)
2881
.666
.696
80.5%
80.1%
.835
.488
Subl->Sub7 (1h)
.230
.455
.543
75.7%
75.4%
.663
.512
Table 9: Subject-adaptive risual-to-fMRI synthesis baseline performance using LinearReg.
H
Additional Results
H.1
Subject-specifie Visual-to-MMRI Synthesis
To evaluate how SynBrain performs under subject-specific modeling settings, we independently train and evaluate SynBrain for four subjects (Sub1, Sub2, Sub5, Sub7). Table TO reports quantitative results across voxel-level accuracy, semantic alignment, and cross-modal retrieval.
Overall, SynBrain achieves strong and consistent performance in all metrics across individuals. These results suggest that SynBrain can effectively model individual fMRI response distributions, producing neural representations that retain both structural fidelity and semantic consistency under subject-specific training.
H.2 Few-shot Subject-adaptive Visual-to-fMRR Synthesis
To comprehensively assess the generalization ability of SynBrain under few-shot settings, we conduct subject-adaptive AisullttofMRi synthesis by transferring models trained on one source subject to novel target subjects using only one hour of adaptation data. While the main paper reports transfers from Subl to Sub2, Sub5, and Sub7, here we extend the evaluation in two directions: (i) expanding to additional subjects and (ii) performing reverse-direction transfers.
(i) Evaluation on Additional Subjects. To examine SynBrain's ability to generalize across more diverse population, we expanded our experiments to include all eight real subjects from the NSD dataset. The main experiments involved four subjects (Sub1. Sub2, Sub5, Sub7), whereas the remaining four (Sub3, Sub4, Sub6, Sub8), who completed fewer sessions, were incorporated for supplementary evaluation. As shown in Table II (Top), SynBrain maintains strong performance across these additional unseen individuals with only one hour of adaptation data, demonstrating robust generalization despite substantial inter-subject neural variability.
Subject Adaptation. We further investigated reverse transfer by adapting models pretrained Sub2, Sub5, and Sub7 to Subl. As presented in Table TT (Bottom), all three source-subject models achieve consistently high semantic-level decoding accuracy (Inception: 88.0-88.9% CLIP: 86.7-86.9% and comparable voxel-level alignment after limited adaptation. Moreover, the MMRI-to-image retrieval accuracy based on the synthesized signals (Syn: 63.5-69.4% closely matches the results observed in the Subl-+Other setting reported the main text. These results confirm that SynBrain enables bidirectional cross-subject adaptation with minimal data, underscoring robustness and adaptability in low-resource, cross-subject generalization scenarios.
(ii) Reverse-Direes
H.3 Data Augmentation for Few-shot IMRI-to-Image Image Decoding
We further evaluate the utility of SynBrain-generatee fMRI signals as a data augmentation strategy for low-resource neural decoding. Beyond Subl (as shown in the main paper), extend our experiments to Sub2, Sub5, and Sub7. For each target subject, we combine their 1-hour real fMRI data with synthetic IMRI signals generated by SynBrain (Subl-SSu -uuuX) using visual stimuli from different,
29

---

Voxel-Level
Semantic-Level (via decuding)
Image Retrieval
Method
MSE 4
Pearson 1
Cosine
Incep 1
CLIP
Eff
SwAV 4
Raw f
Syn* t
SynBrain (Subl)
.079
.715
.769
96.7%
95.9%
.619
.300
94.7%
99.3%
SynBrain (Sub2)
.144
.667
.715
96.2%
94.5%
.637
.364
92.4%
97.1%
SynBruin (Sub5)
.190
.739
.793
96.2%
93.9%
.627
.356
78.0%
86.4%
SynBruin (Sub7)
.151
.626
.680
93.6%
93.0%
.671
.380
73.9%
87.3%
Table 10: Quantitative subject-specific visual-to-IMRI synthesis performance.
Voxel-Leevel
Semantic-Level (via decoding)
Image Retrieval
Method
MSB
Pearson
Cosine
Incep
CLIP
Eff 4
SwAV
Raw 1
Syn
SynBrain (Subl -Sbb3, 1h)
.082
.755
.801
88.1%
85.9%
.781
.464
13.5%
69.1%
SynBrain (Subl -ub4, 1h)
.089
.739
.798
89.0%
86.9%
.759
.455
13.5%
73.1%
SynBrain (Subl -Sub6, Ih)
.067
.727
.785
89.1%
87.1%
.771
.456
16.6%
73.4%
SynBrain (Subl Subb,, th)
.146
.620
.687
85.1%
82.8%
.807
.486
8.9%
69.9%
SynBrain (Sub2 :Sub1, 1h)
.089
.788
88.9 88.9%
86.9%
.778
.448
27.3%
68.4%
SynBruin (Sub5 SSubl 1h)
.086
.91
.752
88, 1%
86.7%
.780
.446
20.7%
63.5%
SynBrain (Sub7-Subb lh)
.086
.685
.746
88.0%
86.7%
.768
.441
23.9%
69.4%
Table 11: Quantitative few-shot visual-to-fMRI synthesis performance on novel subjects.
unseen hour. This augmented dataset is used train the IMRI-to-image decoder (MindEye2+DA4 which is compared against the baseline MindEye2 and state-of-the-ee MindAligner trained only on 1 hour of real data.
As shown in T2] MindEye2+DA consistently outperforms MindEye2 across all subjects. Notable
improvements are observed in high-level semantic metrics (e.g., Inception Score and CLIP) and image-
to-brain retrieval accuracy, suggesting that the generated fMRI signals are more semantically aligned with the intended visual semantics. Moreover, MindEEy22+DA even outperforms MindAligner [II] across all low-level and high-level metrics, reaching state-of-the-art performance in few-shot (1h) EMRI-to-image decoding tasks. However, we also note a slight drop in brain-to-image retrieval accuracy (i.c., retrieving ground truth images given fMRI signals) after augmentation. We hypothesize that this is due to the semantic nature of the synthetic fMRI signals, lacking low-level perceptual details such as pose, background, or texture. While this semantic consistency benefits cross-modal alignment and robustness, may reduce instance instance-specffic discriminability.
Nevertheless, these results confirm that SynBrain-generated fMRI representations are sufficiently real- istie and semantically aligned with visual stimuli to effective training signals for downstream brain decoding models, particularly in few-shot regimes.
Trade-off between Naturalistic and Synthetic Data. Here we conduct additional experiments with extended synthetic data durations (up 32 hours) and observed that the optimal performance is achieved with just 1 hour of data augmentation. As shown in Table [3] adding more synthetic data beyond this point leads diminishing or even negative returns. This suggests trade-offf between real and generated data: while small amount of nigh-quality synthetic data is beneficial, excessive augmentation may introduce distributional mismatch or redundancy, limiting further gains.
H.4 Brain Functional Analysis
To further examine the biological plausibility of SynBrain's generated fMRI representations, perform qualitative analysis of whole-brain activation patterns across subjects. Figures & and p visualize voxel-wise activation maps under both subject-specific and few-shot adaptation settings, respectively.
In Figure 8 we present four representative visual stimuli and compare the generated activation maps across four subjects (Sub1, Sub2, Sub5, Sub7). Despite individual variability in cortical anatomy and voxel organization, SynBrain produces consistent activation distributions across subjects in response to the same stimulus. The decoded images from different subjects also retain semantic consistency, suggesting that the synthesized neural patterns encode shared perceptual features.
Figure further compares activation maps from SynBrain models trained under full-data and few- shot (1-hour) settings. For each stimulus category (airplane, face, giraffe, pizza), we visualize the
30

---

LoveL
High-Level
Retrieval
Method
PixCorr 1
SSIM 1 Alex-21
Alex-5
Incep
CLIIP
BMY
SwAV 4
Image 1
Brain f
MindEye2 (Sub2, 1h)
.200
.433
85.0%
92.1%
81.9%
79.4%
.807
.467
90.5%
67.2%
MindAligner (Sub2. lh)
.218
.426
88.1%
93.3%
84.1%
82.5%
.791
.452
90.0%
85.6%
MindEye2+DA (Sub2, 1h)
.222
.42
88.2%
94.0%
85.0%
83.0%
.781
.443
83.7%
79.3%
MindEye2 (Sub5, Ih)
.175
.405
83.1%
91.0%
84.3%
82.5%
.781
.444
66.9%
47.0%
MindAligner (Sub5, lh)
.177
.409
84.7%
91.6%
84.6%
82.8%
.784
.454
70.6%
66.0%
MindEye2+DA (Sub5, 1h)
.200
86.5%
87.2%
.753
.423
60.7%
MindEye2 (Sub7. 1h)
.170
.408
80.7%
85.9%
74.9%
74.3%
.854
64.4%
37.8%
MindAligner (Sub7. 1h)
.183
.407
81.55%
88.3%
79.9%/
77.8%
.834
.487
64.2%
62.6%
MindEye2+DA (Sub7, 1h)
.189
.409
83.8 %
90.1%
81.7 %
79.3% %
.809
.462
59.6%
57.9%
Table 12: Quantitative jects.
decoding performance comparisons on novel sub-
Low-Level
High-Level
Retrieval
Method
PixCorr
SSIM
Alex-2
Alex-5
Incep
CLIP t Cff v
SwAV
Image
Brain t
MindEye2(Ih)
.235
.288
88.0%
93.3%
83.6%
80.8%
.798
.459
94.0%
77.6%
MindEEyc2((b)+AA((h))
.243
.419
90.1%
95.1%
85.1%
84.7%
.770
.432
87.9%
82.0%
MindEyc2((h))AACC))
.222
.417
88.5%
93.6%
84.9%
82.5%
.795
.450
67.1%
68.8%
(indEEyc2((b+DD((1d))
.229
.416
88.4%
93.4%
83.1%
81.8%
.812
.457
64.8%
63.7%
MindEEc2((h)+4AAG52))
.222
.410
88.4%
93.1%
82.8%
81.8%
.813
.457
61.6%
61.9%
Table 13:
Www-sho.fMRRtto-imee decoding with various hours of synthetic data.
Visual Stimuli
SynDruin IMRI Synthesis
IMRI-to-Imae Visualization
Sub-1
Sub-2
Sub-5
Sub-7
Sub-1
Sub-2
Sub-5
Sub-7
Figure 8: Comparisons of activation maps and fMRI-to-Image visualizations across subjects evoked by the same visual stimuli. All models trained on full data (40h) from specific subjects.
generated activations from the fully trained model (i.e., Sub2, Sub5, Sub7) and the few-shot adapted model (i.e.. Sub2, -ub5, Subl-+Sub7). The results show that even under limited data conditions, the few-shot models can preserve key cptegory-specific activation patterns, such as enhanced responses in occipital and ventral temporal areas for high-level object categories.
These findings suggest that SynBrain not only achieves high-level semantic alignment but also captures spatially meaningful and sategory-sensitive neural patterns that are consistent across subjects under data- limited regimes.
31

---

Visual Stimuli
SynBrain IMRI Synthesis
Airplanc
Subl-Subb
Subl Suib
Subl Suhh?
Facee
Girafie
Pizza
Figure 9: Comparisons of activation maps between full-data (40h) training (i.e., Sub2, Sub5, Sub7) and few-shot (Ih) adaptation (i.e., Subl Sub2, Subl- Subl-+ubl7 across subjects evoked by representative categories of visual stimuli.
32


Reviews:
[{"review_id": "2gDUvRXYhO", "reviewer": "Reviewer_45Ff", "summary": "This current model addresses a one-to-many relationship in stimulus to brain mapping by transforming semantically rich visual stimuli into synthesized neural activity patterns. The authors introduce a non-deterministic model in which both seen and unseen visual stimuli are used to generate synthesized fMRI data. The model performance is reported through a series of comparisons between raw and synthetic voxel activity as well as the resulting generated images' similarity to the original images.", "questions": "Have you considered using a simulation to generate a dataset with more subjects? It is a bit hard to state that the model tested variability across humans (rather than within individuals) with only 4 subjects. \n\nfMRI-to-Image vis -- some of the images generated from the synthesized fMRI capture the gist of the input visual stimulus but have very odd non-sensical distortions to them (e.g. the teddy bear has too many limbs; background vehicles in bus image), how would you explain these distortions? The model is supposed to capture high-level visual representations but seems like it is not reproducing semantic features in the background (vs the foreground).", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "### Pros:\n\n* The current model addresses high variability across humans (and thus across differences in individual visual systems), accounting for variance across physiological and trial-level constraints.\n\n* They employ a novel non-linear transformation of CLIP embeddings into fMRI latent space.\n\n* All together, the model is built to generate fMRI signals that represent high-level visual processing regions.\n\n### Cons:\n\n* (Very, very minor comment) please correct the typo in Figure 3 (Image Retrieval Accuracy).\n\n* The dataset on which the model was tested had very few subjects. The Natural Scenes Dataset only contains 8 subjects in total (albeit several hours of viewing per subject), of which this paper could only utilize 4 subjects due to experiment completion. Please consider a simulation testing variability across more simulated individuals.\n\n* Visual images generated (Figure 4) seem to capture the gist of the input visual stimulus but have some strange distortions.", "was_revised": false, "final_justification": "", "rebuttal": "**W1.** Thanks for your correction.\n\n**W2 & Q1.** Thank you for the suggestion. While simulation-based evaluation is a promising direction, we chose to leave it to future work due to concerns about its **biological validity**. Most simulation methods rely on injecting noise or applying spatial transformations to existing fMRI data, but it remains unclear whether such approaches can faithfully replicate the **complex, subject-specific variability** observed in real neural responses.\n\nAs shown in **Supplementary Figure 3**, even when presented with the same visual stimulus, **different subjects exhibit substantial differences in activation patterns**. This raises concerns about whether current simulation techniques can meaningfully benefit cross-subject modeling without introducing unrealistic assumptions.\n\nIn line with the reviewer’s suggestion to evaluate on a more diverse population, we expanded our experiments to include **all eight real subjects** from the NSD dataset. Four subjects (Sub-1,2,5,7) were used in the main experiments, and the remaining four (Sub-3,4,6,8), who completed fewer sessions, were incorporated as supplementary evaluations to further assess SynBrain’s cross-subject few-shot adaptation capability.\n\nAs shown below, SynBrain achieves strong performance across these unseen individuals with only 1 hour of adaptation data, demonstrating **robust generalization across real, subject-specific neural variability**.\n\n| SynBrain(1h)         | MSE↓ | Pearson↑ | Cosine↑ | Incep↑ | CLIP↑ | Eff↓ | SwAV↓ | Raw↑  | Syn↑  |\n|----------------|-------|------------|-----------|----------|---------|--------|--------|--------|--------|\n| Sub1→Sub3    | 0.082 | 0.755      | 0.801     | 88.1%    | 85.9%   | 0.781  | 0.464  | 13.5%  | 69.1%  |\n| Sub1→Sub4    | 0.089 | 0.739      | 0.798     | 89.0%    | 86.9%   | 0.759  | 0.455  | 13.5%  | 73.1%  |\n| Sub1→Sub6    | 0.067 | 0.727     | 0.785     | 89.1%    | 87.1%   | 0.771  | 0.456  | 16.6%  | 73.4%  |\n| Sub1→Sub8    | 0.146 | 0.620      | 0.687     | 85.1%    | 82.8%   | 0.807  | 0.486  | 8.9%  | 69.9%  |\n| Sub1→Sub2   | 0.160   | 0.619   | 0.675  | 89.2% | 88.1% | 0.751 | 0.431 | 19.5% | 67.4%  |\n| Sub1→Sub5   | 0.224   | 0.704   | 0.765  | 89.2% | 88.0% | 0.749 | 0.432 | 16.8% | 54.8%  |\n| Sub1→Sub7   | 0.151   | 0.630   | 0.679  | 86.8% | 84.7% | 0.783 | 0.453 | 13.2% | 76.5%  |\n    \n\n**W3 & Q2.** Thanks for your comment. We agree that some images generated from the synthesized fMRI exhibit noticeable distortions, particularly in background details. These artifacts primarily stem from the limited capacity of the fMRI-to-image decoder used in our pipeline—namely, an **unrefined version of MindEye2** [1]. As shown in the original MindEye2 paper, even with ground-truth fMRI input, the reconstructed images can retain overall semantic content while exhibiting unnatural elements. This issue could be mitigated by replacing the decoder with a **refined version of MindEye2** (i.e., by guiding the unrefined reconstruction through base SDXL with image-to-image generation), which offers improved perceptual quality and semantic alignment.\n\nIn our work, however, the **decoder** is used **solely as a proxy** to assess whether the synthesized fMRI preserves meaningful high-level representations. In fact, due to **its own semantic decoding limitations**, the reported semantic-level scores may **underestimate** the true alignment between the synthesized fMRI and the corresponding visual content.\n\n[1] Scotti P S, Tripathy M, Villanueva C K T, et al. MindEye2: Shared-Subject Models Enable fMRI-To-Image With 1 Hour of Data. ICML 2024.\n\nWe thank the reviewer again for the valuable feedback. We hope our response adequately addresses your comments. If so, we hope you could consider a more positive evaluation of our work. If there are any remaining concerns, we would be happy to further address them during the discussion phase."}, {"review_id": "tEZN2vLOKD", "reviewer": "Reviewer_sdRq", "summary": "This paper introduces SynBrain, a generative framework that operates under the brain decoding scenario. Experiments show that SynBrain generates more coherent data compared to the baseline and can improve existing decoding models as a data augmentation engine under limited-data scenarios.", "questions": "- What kind of fMRI input was used? The \"nsdgeneral\" region, like previous works such as MindEye\n- For the data augmentation experiments in table 2, did the authors try traditional data augmentation strategies as a comparison?\n- Experimental details for figure 3: \n    - Are the experiments about just swapping the brain encoder/decoder from the pipeline in figure 2?\n    - Is the MSE loss evaluating only the autoencoder, while the other losses evaluate the synthesis pipeline?\n- I'm not sure if it's a good idea to introduce two (possibly competing) objectives that directly optimize the latent z in equation 4. Is it really necessary to introduce the SoftCLIP loss to the latent, instead of some other part of the pipeline, when it is already conditioned by the KL loss? I understand that outright removing the SoftCLIP loss is harmful, as seen in table 3, but can't it be introduced in some other part instead of being directly fed into the latent z that's supposed to follow a Gaussian distribution?\n- On a related note, how do the KL loss and SoftCLIP loss behave during training? Do $\\lambda_{KL}$ and $\\lambda_{CLIP}$ keep the two losses in the general ballpark during training? Do the latent vectors of the training samples properly follow a Gaussian distribution after training?", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 4, "originality": 3, "strengths_and_weaknesses": "## Strengths\n- The proposed data synthesis pipeline outperforms the shown baselines.\n- The proposed method can also be utilized to improve existing decoding models as a data augmentation strategy under limited-data scenarios.\n\n## Weaknesses\n- There are some concerns regarding the theoretical motivation of the proposed loss function (Eq. 4); more on this in the questions section.\n- Details on some of the experiments, mainly figure 3, seem to be absent.", "was_revised": true, "final_justification": "They have answered most of my questions and concerns. Many of them stem from the confusion I had while reading the manuscript, but I expect the authors to revise the manuscript to reflect my questions and confusion in the final version.", "rebuttal": "**Q1.** Yes, we used the “nsdgeneral” cortical region as fMRI input, following previous works like MindEye.\n\n\n**Q2.** Thank you for the insightful question. In our experiments, the fMRI data generated by SynBrain was used as an additional augmentation strategy to further enhance the performance of MindEye2 in the 1-hour few-shot adaptation setting. It’s worth noting that MindEye2 already incorporates standard data augmentation techniques (i.e., image distortions and MixUp) during training. **Our intention was not to compare SynBrain with traditional augmentation methods, but rather to build upon them.** Specifically, we aimed to validate that SynBrain-generated fMRI can effectively complement real fMRI data, helping to alleviate the challenge of data scarcity and improve model generalization in low-resource scenarios.\n\n**Q3.** Thank you for the thoughtful questions, and we apologize for the lack of clarity in the current version of Figure 3. To clarify, **Figure 3 corresponds exclusively to Stage-1 (BrainVAE) in Figure 2**, which focuses on training the BrainVAE module. The experiments in Figure 3 investigate **the effect of replacing the encoder and decoder architecture within Stage-1**. Specifically, we compare our proposed BrainVAE with two alternative designs—MLP-AE and MLP-VAE—while keeping the loss functions and training setup identical. As defined in **Equation (4)**, all models are optimized using **KL loss, MSE loss, and CLIP loss**. The only difference lies in the encoder-decoder architecture.\n\nRegarding the second question, all loss terms are used to evaluate the Stage-1 (BrainVAE) only. The MSE loss measures **voxel-level reconstruction quality**, while the CLIP loss evaluates the **semantic consistency between the brain latent $z$ and the image embedding $z_{CLIP}$**. Figure 3 (Right) shows the validation loss curves (MSE and CLIP) during Stage-1 training for each architecture. To further assess the quality of the brain latent space, we also conduct **Brain-Image Retrieval between $z$ and $z_{CLIP}$** in Stage-1. Higher retrieval accuracy reflects stronger semantic alignment.\n\nFinally, we emphasize that **Figure 3 is entirely unrelated to Stage-2 (S2N Mapper) in Figure 2**; the experiments and comparisons are **solely focused on Stage-1 (BrainVAE)**.\n\n**Q4.** Thank you for your thoughtful and detailed question. In our design, the KL loss and SoftCLIP loss are applied to the latent variable z with **complementary purposes**: the KL loss encourages smoothness and sampleability in the latent space, while the SoftCLIP loss promotes semantic alignment between brain and image modalities.\n\nImportantly, we intentionally **downweight the KL loss**, allowing it to function primarily as a **soft regularizer** rather than strictly enforcing a standard Gaussian prior. This **relaxed constraint** gives the latent space flexibility to encode semantically aligned structure, while still preserving the stochastic sampling capability that underlies variational generation.\n\nAs shown in **Table 3**, removing either KL (Variation Sampling) or CLIP (Contrastive Learning) objective significantly degrades performance, underscoring their necessity. Furthermore, **Supplementary Figure 2** demonstrates that their combination enables **semantically consistent sampling** from the latent space—achieving the one-to-many generation behavior that BrainVAE is designed to support.\n\nThis design choice is also supported by prior work such as **VAVAE** [1], which demonstrated the effectiveness of combining semantic constraints from visual encoders (e.g., CLIP, DINOv2) with variational modeling. \n\n[1] Yao, J., Yang, B., & Wang, X. Reconstruction vs. generation: Taming optimization dilemma in latent diffusion models. CVPR 2025 Oral.\n\n**Q5.** Thank you for the insightful follow-up. As shown in Figure 3, both the MSE loss and CLIP loss decrease steadily over the first 15 epochs and then stabilize, indicating that the model converges in terms of both voxel-level reconstruction and semantic alignment. The KL loss follows a similar trajectory, converging within 15 epochs.\n\nTo balance the three objectives, we apply appropriate weighting: $\\lambda_{KL}=0.001$, $\\lambda_{CLIP} = 1000$. After weighting, the KL loss remains at the order of $10^1$, MSE loss decreases from $10^3$ to $10^2$ over training, and CLIP loss stays around the order of $10^3$. This setup ensures that each loss term contributes in a controlled and complementary manner throughout training. In particular, the CLIP loss plays a more influential role in the early stages, helping guide the latent space toward semantic alignment, while the KL loss provides soft regularization to support sampling.\n\nImportantly, our goal is not for the latent vectors to strictly follow a standard Gaussian distribution, but to remain sufficiently regularized to support stochastic sampling. As further illustrated in **Supplementary Figure 2**, SynBrain can perform stochastic sampling in the latent space (via noise injection) and still generate semantically consistent fMRI signals. This capability reflects the core design principle of BrainVAE: to **balance variational sampling with meaningful semantic structure**, rather than enforce a rigid Gaussian prior.\n\nWe thank the reviewer again for the valuable feedback. We hope our response adequately addresses your comments. If so, we hope you could consider a more positive evaluation of our work. If there are any remaining concerns, we would be happy to further address them during the discussion phase."}, {"review_id": "S9iF9gWLtU", "reviewer": "Reviewer_fiTc", "summary": "This paper proposes SynBrain, a generative framework for visual-to-fMRI synthesis that models neural responses as semantic-conditioned probability distributions. The key innovation lies in addressing the inherent one-to-many mapping from visual stimuli to neural responses through probabilistic modeling. SynBrain consists of two main components: (1) BrainVAE, a variational autoencoder that models fMRI signals as continuous probability distributions conditioned on visual semantics, and (2) S2N (Semantic-to-Neural) Mapper, a Transformer-based module that maps CLIP visual embeddings to the fMRI latent space. The authors evaluate their approach on the Natural Scenes Dataset (NSD) with 4 subjects, demonstrating superior performance in subject-specific visual-to-fMRI synthesis and effective few-shot adaptation to novel subjects.", "questions": "1. How does the computational cost of SynBrain compare to deterministic baselines? The paper mentions faster training but doesn't provide detailed timing comparisons or discussions. Is this impacted by the necessary computing hardware to train and run their method? The authors note that they use 4 A100's to train their method, it is possible to train it on a single GPU or with reduced hardware requirements? Does  the variational approach have meaningful differences in inference efficiency?\n\n2. The authors claim that synthesized fMRI signals achieve higher retrieval accuracy (92.5%) than raw fMRI (84.8%). Could this indicate that the model is learning to enhance or denoise the signals rather than faithfully reproduce the noisy properties of neural responses? It seems to me that one of the primary advantages of a variational system is the ability to more accurately model the noise properties of real neural data, what do the authors make of this point in the context of the \"better\" retrieval scores?\n\n3. How sensitive is the approach to the choice of visual encoder (CLIP)? Have the authors experimented with other pre-trained vision models?\n\n4. The few-shot adaptation experiments use subjects from the same dataset. How would the approach perform when adapting to subjects scanned with different protocols or equipment? Transferring to new subjects with minimal data in new datasets would be of significant interest to downstream applications of the approach and would boot the significance of the work.\n\n5. Could the authors provide more analysis on what specific types of neural variability the model captures versus what it might be missing?\n\n6. On line 171 the authors note that MindEye uses a diffusion transformer, while the MindEye paper actually uses a *diffusion prior*. This detail should be corrected for the camera ready version.", "limitations": "The authors adequately address limitations in the conclusion, including reliance on pre-trained vision models and incomplete modeling of variance sources. However, they could expand on computational requirements, unknown dataset transferability, and any scalability concerns.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n\n**Quality:**\n- The paper addresses a fundamental challenge in computational neuroscience by explicitly modeling the one-to-many nature of visual-to-neural mappings, which is biologically motivated and theoretically sound.\n- The experimental design is comprehensive, evaluating across voxel-level, semantic-level, and retrieval metrics, providing multi-faceted validation of the approach.\n- The ablation studies clearly demonstrate the contribution of each component (variational sampling, contrastive learning, S2N Mapper).\n- The architecture design is well-motivated, particularly the use of convolutional and attention layers in BrainVAE to address instability issues in MLP-based VAEs.\n\n**Clarity:**\n- The paper is generally well-written with clear motivation and methodology sections.\n- The figures effectively illustrate the approach and results, particularly Figure 2 showing the overall framework.\n- The experimental setup is clearly described with sufficient implementation details for reproducibility.\n\n**Significance:**\n- The work addresses an important limitation in existing deterministic approaches to brain encoding, offering a more biologically plausible framework.\n- The results demonstrate practical benefits for data augmentation in few-shot fMRI-to-image decoding scenarios.\n- The findings about cross-trial and cross-subject functional consistency provide insights into neural variability patterns.\n\n**Originality:**\n- The combination of probabilistic modeling with semantic conditioning for fMRI synthesis is novel and well-motivated.\n- The S2N Mapper design offers a more efficient alternative to diffusion-based approaches used in prior work.\n- The framework successfully balances modeling neural variability while preserving functional consistency.\n\n**Weaknesses:**\n\n**Quality:**\n- The evaluation is limited to a single dataset (NSD). While this is the largest and highest quality dataset available, validation on additional datasets would strengthen the claims, and I am particulary interested to see how the variational approach proposed would handle lower quality and smaller datasets.\n- The comparison baseline is primarily limited to MindSimulator. The paper would benefit from comparisons to other established image-to-fMRI encoding models, such as the GNet model introduced in \"Brain-optimized deep neural network models of human visual areas learn non-hierarchical representations\" (St-Yves et al., 2023) and the Transformer-based approach from \"Predicting Human Brain States with Transformer\" (Sun et al., 2024). These comparisons would better contextualize SynBrain's performance relative to the broader landscape of visual encoding models, even if these baselines are deterministic.\n\n**Clarity:**\n- Some technical details about the BrainVAE architecture could be clearer, particularly the specific design choices and architectural details for the convolutional and attention components. This could have been bolstered by making code accessible to reviewers as well.\n\n**Significance:**\n- While the results are promising, the practical impact may be limited by the computational overhead of the probabilistic approach compared to deterministic methods. Can the authors comment on the computational efficiency of the method compared to existing approaches?\n- The few-shot adaptation results, while encouraging, are evaluated on relatively similar subjects from the same dataset and the same scanner. Without cross-dataset adaptation results it is difficult to quantify the true impact of this result.\n\n**Originality:**\n- While the overall approach is novel, individual components (VAEs, contrastive learning, Transformers) are well-established techniques.\n- The novelty lies primarily in their combination and application to fMRI synthesis rather than fundamental algorithmic innovations.", "was_revised": true, "final_justification": "I am impressed by the depth of their rebuttal, and consider the majority of my concerns addressed. In particular, the drastic improvements fine tuning across datasets, the massive boost in inference efficiency, additional baselines, and increased clarity into the training efficiency of their model I think are all important items that boost the significance of their submission. I will increase my score to 5, as I believe the paper represents a high-impact improvement to the paradigm and performance of visual encoding models for fMRI.", "rebuttal": "**W-Clarity1.** Thanks for the feedback. We will clarify the architectural details of BrainVAE in the revised manuscript. While the code cannot be shared during rebuttal for anonymity, it will be released upon acceptance to ensure reproducibility.\n\n**W-Quality2.** Thank you for the suggestion. Note that Predicting Human Brain States with Transformer (Sun et al., 2024) focuses on resting-state fMRI and aims to predict future brain states based on past activity. As such, it does not fall within the scope of **visual-to-fMRI encoding** and is not directly comparable to our setting. Hence, we have included additional comparisons with two representative baselines: (1) **GNet** (St-Yves et al., 2023), and (2) **Linear Regression**, a classical deterministic voxel-wise visual encoding model, using a simple two-layer linear architecture with a 4,096-dimensional hidden space.\n\nAs shown below, SynBrain achieves consistently **superior results across multiple levels**, highlighting its effectiveness in fMRI synthesis with both voxel-level structure and semantic consistency.\n\n| Method     | MSE↓  | Pearson↑ | Cosine↑ | Incep↑ | CLIP↑ | Eff↓  | SwAV↓ | Raw↑  | Syn↑  |\n|------------|--------|-----------|-----------|----------|---------|--------|---------|--------|--------|\n| SynBrain   | **0.079**  | **0.715** | **0.769** | **96.7%** | **95.9%** | **0.619** | **0.350** | **94.7%** | **99.3%** |\n| LinearReg  | 0.102  | 0.676     | 0.693     | 85.8%   | 83.7%   | 0.759  | 0.454   | –      | –      |\n| GNet       | 0.092 | 0.707     | 0.740     | 87.7%   | 85.0%   | 0.730  | 0.428   | –      | –      |\n\n**Q1.** \n\n**Training Efficiency.**\nThe faster training of BrainVAE is mainly due to its **rapid convergence** within fewer epochs. Under the **same total batch size** of 24, BrainVAE completes 15 epochs in ~2 hours (4×A100-40G), while MLP-AE requires 50 epochs and ~5 hours (1×A100-40G), yet still underperforms BrainVAE across key performance metrics. \nNote that under the same total batch size, multi-GPU training introduces extra overhead due to gradient synchronization across devices, resulting in a slightly longer per-epoch time compared to single-GPU training. Nonetheless, BrainVAE significantly shortens the overall training time by converging within far fewer epochs.\n\n**Hardware Flexibility.**\nWhile the results reported in the main paper use 4×A100-40G GPUs (with downsampling blocks = 1, latent dim = 4096), we also experimented with **lighter variants** of BrainVAE, as reported in **Supplementary Table 1**. For example, setting the number of **downsampling blocks to 3 (latent dim = 1024)** enables faster training on a **single** A100-40G GPU, with only minor performance degradation (-1.5% in Incep and -1.4% in CLIP). Even so, it still significantly outperforms MindSimulator (5-Trial) by 2.4% (Incep) and 4.7% (CLIP), demonstrating the scalability of SynBrain to more constrained computational environments.\n\n**Inference Efficiency.**\nThe observed difference in inference efficiency is not inherently due to the variational approach itself, but rather stems from the design of architecture used in the Visual-to-fMRI synthesis process. Specifically, MindSimulator employs a multi-step diffusion-based method (Diffusion Transformer, DiT), while SynBrain uses **a one-step S2N mapper for direct sampling**. This leads to a substantial difference in inference time as shown below.\n\n| Method              | Inference Time(s) |\n|:------------------- |:-----------------:|\n| SynBrain (S2N)      |       0.059       |\n| MindSimulator (DiT) |      31.845       |\n\n**Q2.** **Semantic Consistency vs. Voxel-level Fidelity**\n\nThanks for your comment. While BrainVAE is capable of modeling the noise characteristics inherent in neural signals through its latent space, it is **not intended to faithfully reconstruct the full voxel-level complexity** of raw fMRI signals. Instead, by integrating visual semantic constraints during training, the model **prioritizes semantic consistency over voxel-wise fidelity**, resulting in synthesized signals that retain relevant variability while **suppressing irrelevant noise**.\n\nThe improved retrieval accuracy reflects this design choice. Rather than contradicting the variational framework, it highlights the capability of BrainVAE to **balance biologically plausible variability with task-relevant constraints**, enabling effective sampling and generation aligned with perceptual semantics. As shown in **Supplementary Figure 2**, fMRI synthesis via multiple stochastic passes of BrainVAE exhibit voxel-level variability, yet remain **semantically consistent across samples**. \n\n\n**Q3.** **Vision Model Comparison:** \n\nThanks for your thoughtful question. We evaluated the impact of visual encoder choice by replacing CLIP in SynBrain with two advance visual foundation models, **DINOv2** and **MAE**. As shown in the table below, CLIP achieves the best overall performance across reconstruction, voxel-level correlation, and semantic alignment. We attribute this to its vision-language contrastive objective, which provides stronger semantic supervision compared to purely visual models like DINOv2 and MAE. Nonetheless, the relatively strong performance of these alternatives indicates that SynBrain remains robust across different encoder choices.\n\nWe also include a comparison with MindSimulator (5-Trial), which likewise uses CLIP as its visual encoder. Notably, MindSimulator averages results over five generated fMRI trials to improve performance, whereas SynBrain achieves superior results with just a single trial. This suggests that SynBrain benefits not only from a strong encoder but also from more effective model design and training strategy.\n\n| Method                  | VisEncoder | MSE↓  | Pearson↑ | Cosine↑ | Incep↑ | CLIP↑  | Eff↓  | SwAV↓ | Raw↑  | Syn↑  |\n|-------------------------|----------------|-------|----------|---------|--------|--------|-------|--------|-------|-------|\n| MindSimulator (5-Trial) | CLIP           | 0.417 | 0.326    | -       | 92.8%  | 89.8%  | 0.714 | 0.402  | -     | -     |\n| SynBrain (1-Trial)      | CLIP           | 0.079 | 0.715    | 0.769   | 96.7%  | 95.9%  | 0.619 | 0.350  | 94.7% | 99.3% |\n|                         | DINOv2         | 0.089 | 0.679    | 0.728   | 92.6%  | 91.8%  | 0.701 | 0.375  | 76.3% | 87.6% |\n|                         | MAE            | 0.097 | 0.676    | 0.737   | 91.0%  | 90.5%  | 0.705 | 0.389  | 68.6% | 83.7% |\n\n\n**W1&Q4.** **Cross-dataset Adaptation** \n\nThanks for the suggestions. To tackle this concern, we conducted extra experiments on the GOD [1] dataset, which differs from NSD in both scanning protocol and data quality. Each GOD subject contains only 1,200 training images collected in a single session and 50 out-of-class test images, offering a challenging setting for cross-dataset adaptation.\n\nGOD subjects used the Visual ROI defined by MindVis (4,623 voxels), and for consistency, we employed a SynBrain variant trained on the NSD Visual ROI (4,657 voxels), instead of the standard NSDGeneral ROI (15,724 voxels, from MindEye). This variant features 3 downsampling blocks ($dim=512$) that enable single-GPU training.\n\nWe compared two conditions on GOD: (1) training SynBrain from scratch, and (2) adapting a SynBrain model pretrained on NSD (Sub1). As shown in the table below, the **pretrained model substantially outperforms the from-scratch baseline** across all metrics. \n\nNote that, in this case, **voxel-level correlations remain relatively low**—primarily due to the **poor SNR** (Signal-Noise Ratio) of GOD (~0.1, as reported in MindVis [2]). However, the model achieves **substantial gains at the semantic level**. This suggests that under such noisy conditions, high voxel-wise correlation may reflect overfitting to noise rather than capturing meaningful neural structure. SynBrain, by contrast, prioritizes semantic alignment and achieves $68.0$% **50-way Top-1 retrieval accuracy** using synthesized fMRI, outperforming Mind-Vis (pretrained on HCP), which reaches 23.9% with raw fMRI. These results highlight SynBrain’s effectiveness in transferring to small-scale, low-SNR datasets.\n\n\n| Method         | Pretrained   | MSE↓   | Pearson↑ | Cosine↑ | Raw↑  | Syn↑  |\n|----------------|--------------|---------|-----------|-----------|--------|--------|\n| Mind-Vis       | HCP          | –       | –         | –         | 23.9%  | –      |\n| SynBrain-GOD   | –            | 18.283  | 0.104     | 0.119     | 6.0%   | 12.0%  |\n| SynBrain-GOD   | NSD(Sub1)   | 6.845   | 0.151     | 0.176     | 13.0%  | 68.0%  |\n\n\n[1] Horikawa, T. & Kamitani, Y. Generic decoding of seen and imagined objects using hierarchical visual features. Nature Communications 2017.\n\n[2] Chen Z, et al. Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding. CVPR 2023.\n\n**Q5.** Thank you for the valuable question. BrainVAE is designed to capture **stimulus-driven, semantically relevant variability** by modeling **trial-level one-to-many mappings**—that is, the distribution of neural responses elicited by repeated or similar visual stimuli. This allows the model to learn not just a single canonical response per stimulus, but a functionally consistent response space shaped by perceptual semantics. While the model can, in principle, capture broader sources of variability, **subject-specific idiosyncrasies and spontaneous background activity** are typically down-weighted, as they are less aligned with visual semantics. \n\n**Q6.** Thank you for pointing this out. We acknowledge the terminology difference and will revise the description accordingly in the camera-ready version. \n\nWe thank the reviewer again for the valuable feedback. We hope our response adequately addresses your comments. If so, we hope you could consider a more positive evaluation of our work. If there are any remaining concerns, we would be happy to further address them during the discussion phase."}, {"review_id": "rWMjiULUGc", "reviewer": "Reviewer_DDYY", "summary": "This paper presents SynBrain, a generative model that learns to synthesize fMRI responses from visual input using a probabilistic approach. The key idea is to model brain responses as distributions rather than fixed outputs, better reflecting biological variability. The method performs well and shows good potential for few-shot adaptation.", "questions": "In BrainVAE, fMRI responses are modeled as a latent probability distribution z, capturing the one-to-many mapping. However, in the S2N Mapper, CLIP features are mapped deterministically to a latent feature, resembling a one-to-one mapping. Does this not contradict the biological argument that visual-to-neural mappings are inherently one-to-many? Could the authors clarify whether this deterministic mapping risks averaging out noise or inter-trial variability, and what impact this has on downstream synthesis quality and biological realism?", "limitations": "See weaknesses.", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n1. The use of a semantic-conditioned probabilistic framework to model the one-to-many mapping from visual stimuli to neural responses is both biologically plausible and methodologically sound. This explicitly accounts for trial-level noise, attentional fluctuations, and inter-subject variability.\n2. The proposed method achieves state-of-the-art performance in visual-to-fMRI encoding across several settings.\n3. The generated neural signals are shown to be biologically interpretable.\n\nWeaknesses:\n1. The main experiments are conducted on a single-subject setting. Given the increasing importance of multi-subject modeling in this field, it would strengthen the paper to explore whether incorporating data from multiple source subjects could further improve generalization. For instance, in Table 1, the few-shot adaptation experiments (1→2/5/7) could be extended to include a multi-subject pretraining setting (e.g., 125→7).\n2. The few-shot adaptation experiments lack comparison to suitable baselines. For example, could MindSimulator be fine-tuned on a new subject in a similar few-shot setting, and how would it compare?\n3. In Table 2, the benefit of data augmentation with MindEye2(1h)+DA(1/2/4h) is shown, but the upper bound of the benefit remains unclear. It would be helpful to analyze how much synthetic data is optimal and whether the performance saturates with more hours of augmented data.", "was_revised": true, "final_justification": "The authors have addressed my concerns in the rebuttal.", "rebuttal": "**W1.** Thank you for the thoughtful suggestion. We conducted 1-hour novel-subject adaptation experiments using multi-subject pretraining (e.g., Sub125→Sub7, Sub257→Sub1) and observed modest performance gains compared to single-subject pretraining (e.g., Sub1→Sub7, Sub2→Sub1), as shown below.  \n\nHowever, these improvements required deeper Transformer architectures (8→12 layers) and extended training (50k→200k steps), substantially increasing computational cost. In contrast, single-subject pretraining followed by 1-hour adaptation offers a more efficient trade-off between performance and training resources.\n\n| SynBrain (1h-Adapt)    | MSE↓  | Pearson↑ | Cosine↑ | Incep↑ | CLIP↑ | Eff↓  | SwAV↓ | Raw↑  | Syn↑  |\n|--------------------|-------|----------|---------|--------|--------|--------|--------|--------|--------|\n| Sub1→Sub7          | 0.151 | 0.630    | 0.679   | 86.8%  | 84.7%  | 0.783 | 0.453 | 13.2% | 76.5% |\n| Sub125→Sub7        | 0.137 | 0.658    | 0.695   | 87.2%  | 85.0%  | 0.795 | 0.449 | 14.2% | 78.3% |\n| Sub2→Sub1          | 0.089 | 0.689    | 0.748   | 88.9%  | 86.9%  | 0.778 | 0.448 | 27.3% | 68.4% |\n| Sub257→Sub1        | 0.080 | 0.693    | 0.774   | 89.4%  | 87.3%  | 0.773 | 0.452 | 31.6% | 71.7% |\n\n**W2.** Thank you for the suggestion. Taking MindSimulator as a cross-subject adaptation baseline is limited by two practical constraints. i) **MindSimulator is not open-sourced** and incorporates several **hand-crafted** components (e.g., resting-state initialization, custom noise injection), making it **difficult to reproduce**; ii) MindSimulator’s Autoencoder is **subject-specific** and tightly coupled to **fixed voxel dimensions**, rendering it incompatible with cross-subject scenarios where ROI sizes vary. These limitations prevent a direct and meaningful comparison between SynBrain and MindSimulator in the context of cross-subject adaptation.\n\nTo establish a transparent and reliable baseline, we implemented **Linear Regression**, the most classical encoding model, using a simple two-layer linear architecture. The first layer maps CLIP features to a 4,096-dimensional hidden space, and the second layer projects to subject-specific voxel responses. During few-shot adaptation, we freeze the first layer and train only the second layer using 1-hour data from the target subject. As shown in the results below, **SynBrain consistently outperforms Linear Regression** across voxel-wise correlation, semantic alignment, and image retrieval accuracy, demonstrating strong generalization and cross-subject transferability under data-limited scenarios.\n\n| Method           | 1h-Adapt | MSE↓ | Pearson↑ | Cosine↑ | Incep↑ | CLIP↑ | Eff↓ | SwAV↓ | Raw↑ | Syn↑ |\n|------------------|------------|--------|------------|-----------|-----------|---------|--------|----------|--------|--------|\n| SynBrain         | Sub1→Sub2  | 0.160  | 0.619      | 0.675     | 89.2%    | 88.1%  | 0.751  | 0.431    | 19.5%  | 67.4%  |\n|                  | Sub1→Sub5  | 0.224  | 0.704      | 0.765     | 89.2%    | 88.0%  | 0.749  | 0.432    | 16.9%  | 54.8%  |\n|                  | Sub1→Sub7  | 0.151  | 0.630      | 0.679     | 86.8%    | 84.7%  | 0.784  | 0.453    | 13.2%  | 76.5%  |\n| LinearReg| Sub1→Sub2  | 0.228  | 0.482      | 0.550     | 76.6%    | 76.7%  | 0.861  | 0.514    |   –    |   –    |\n|                  | Sub1→Sub5  | 0.281  | 0.616      | 0.696     | 80.5%    | 80.1%  | 0.835  | 0.488    |   –    |   –    |\n|                  | Sub1→Sub7  | 0.230  | 0.475      | 0.543     | 75.7%    | 75.4%  | 0.863  | 0.512    |   –    |   –    |\n\n\n**W3.** Thank you for the insightful question. We conducted additional experiments with extended synthetic data durations (up to 32 hours) and observed that the **optimal performance is achieved with just 1 hour of data augmentation**. As shown below, adding more synthetic data beyond this point leads to diminishing or even negative returns.\n\nThis suggests a **trade-off between real and generated data**: while a small amount of high-quality synthetic data is beneficial, excessive augmentation may introduce distributional mismatch or redundancy, limiting further gains.\n    \n| Method                 | PixCorr↑ | SSIM↑ | Alex2↑ | Alex5↑ | Incep↑ | CLIP↑ | Eff↓ | SwAV↓ | Image↑ | Brain↑ |\n|------------------------|-----------|--------|---------|---------|----------|---------|--------|---------|----------|----------|\n| MindEye2(1h)    | 0.235     | **0.428**  | 88.0%   | 93.3%   | 83.6%    | 80.8%   | 0.798  | 0.459   | **94.0%**    | 77.6%    |\n| MindEye2(1h)+DA(1h)    | **0.243**     | 0.419  | **90.1%**   | **95.1%**   | **85.1%**    | **84.7%**   | **0.770**  | **0.432**   | 87.9%    | **82.0%**    |\n| MindEye2(1h)+DA(8h)    | 0.222     | 0.417  | 88.5%   | 93.6%   | 84.9%    | 82.5%   | 0.795  | 0.450   | 67.1%    | 68.8%    |\n| MindEye2(1h)+DA(16h)   | 0.229     | 0.416  | 88.4%   | 93.4%   | 83.1%    | 81.8%   | 0.812  | 0.457   | 64.8%    | 63.7%    |\n| MindEye2(1h)+DA(32h)   | 0.222     | 0.410  | 88.4%   | 93.1%   | 82.8%    | 81.8%   | 0.813  | 0.457   | 61.6%    | 61.9%    |\n    \n**Q1.** We thank the reviewer for this insightful question. While the S2N Mapper adopts a deterministic mapping from CLIP features to BrainVAE’s latent space, this design does not contradict the biological one-to-many nature of visual-to-neural mappings.\nIn the **first stage**, BrainVAE is trained to capture trial-level variability by associating each visual input with a latent distribution, **whose center captures the most salient and semantically consistent neural features across trials**. By conditioning on this center and performing **stochastic sampling**, BrainVAE can generate multiple plausible fMRI responses that are semantically aligned yet voxel-wise diverse, faithfully reflecting the one-to-many nature of neural encoding.\n\nIn the **second stage**, the S2N Mapper is optimized to align each CLIP embedding with this **semantic anchor**—the **center of the latent distribution**—rather than with any specific trial or empirical mean. Its objective is to map visual features to the **most representative position** in latent space, while **delegating the modeling of trial-level variability to the stochastic sampling process**. This design ensures that variability is not averaged out, but reintroduced in a controlled, biologically grounded manner.\n\nAs shown in **Supplementary Figure 2**, repeated synthesis from the same visual input yields fMRI signals that are **voxel-wise diverse but semantically stable**. This confirms that our framework supports semantic-consistent sampling and generation, effectively balancing biological realism with representational precision.\n\nWe thank the reviewer again for the valuable feedback. We hope our response adequately addresses your comments. If so, we hope you could consider a more positive evaluation of our work. If there are any remaining concerns, we would be happy to further address them during the discussion phase."}]

Author rebuttal:

Meta-review: This paper introduces a generative model for neural responses called "SynBrain".  The model consists of a VAE and a semantic-to-neural mapping, which allows it to synthesizes high-quality fMRI signals in response to semantic inputs, and achieves high performance using few-shot learning on novel subjects. The problem setting is important and the results are impressive. The reviewers were unanimous in their assessment that the paper makes a worthwhile contribution to the literature and should be accepted.  Congratulations! Please be sure to address all reviewer comments and criticisms in the final manuscript.
Decision: Accept (poster)