Title: SynBrain: Enhancing Visual-to-fMRI Synthesis via Probabilistic Representation Learning
Venue: NeurIPS 2025
Accepted: yes
Keywords: Brain encoding

Probabilistic modeling

Cross-modal generation
Abstract: Deciphering how visual stimuli are transformed into cortical responses is a fundamental challenge in computational neuroscience. This visual-to-neural mapping is inherently a one-to-many relationship, as identical visual inputs reliably evoke variable hemodynamic responses across trials, contexts, and subjects. However, existing deterministic methods struggle to simultaneously model this biological variability while capturing the underlying functional consistency that encodes stimulus information. To address these limitations, we propose SynBrain, a generative framework that simulates the transformation from visual semantics to neural responses in a probabilistic and biologically interpretable manner. SynBrain introduces two key components: (i) BrainVAE models neural representations as continuous probability distributions via probabilistic learning while maintaining functional consistency through visual semantic constraints; (ii) A Semantic-to-Neural Mapper acts as a semantic transmission pathway, projecting visual semantics into the neural response manifold to facilitate high-fidelity fMRI synthesis. Experimental results demonstrate that SynBrain surpasses state-of-the-art methods in subject-specific visual-to-fMRI encoding performance. Furthermore, SynBrain adapts efficiently to new subjects with few-shot data and synthesizes high-quality fMRI signals that are effective in improving data-limited fMRI-to-image decoding performance. Beyond that, SynBrain reveals functional consistency across trials and subjects, with synthesized signals capturing interpretable patterns shaped by biological neural variability. 
Our code is available at https://github.com/MichaelMaiii/SynBrain.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents SynBrain, a generative model that learns to synthesize fMRI responses from visual input using a probabilistic approach. The key idea is to model brain responses as distributions rather than fixed outputs, better reflecting biological variability. The method performs well and shows good potential for few-shot adaptation.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The use of a semantic-conditioned probabilistic framework to model the one-to-many mapping from visual stimuli to neural responses is both biologically plausible and methodologically sound. This explicitly accounts for trial-level noise, attentional fluctuations, and inter-subject variability.\n2. The proposed method achieves state-of-the-art performance in visual-to-fMRI encoding across several settings.\n3. The generated neural signals are shown to be biologically interpretable.\n\nWeaknesses:\n1. The main experiments are conducted on a single-subject setting. Given the increasing importance of multi-subject modeling in this field, it would strengthen the paper to explore whether incorporating data from multiple source subjects could further improve generalization. For instance, in Table 1, the few-shot adaptation experiments (1→2/5/7) could be extended to include a multi-subject pretraining setting (e.g., 125→7).\n2. The few-shot adaptation experiments lack comparison to suitable baselines. For example, could MindSimulator be fine-tuned on a new subject in a similar few-shot setting, and how would it compare?\n3. In Table 2, the benefit of data augmentation with MindEye2(1h)+DA(1/2/4h) is shown, but the upper bound of the benefit remains unclear. It would be helpful to analyze how much synthetic data is optimal and whether the performance saturates with more hours of augmented data.\n\nQUESTIONS:\nIn BrainVAE, fMRI responses are modeled as a latent probability distribution z, capturing the one-to-many mapping. However, in the S2N Mapper, CLIP features are mapped deterministically to a latent feature, resembling a one-to-one mapping. Does this not contradict the biological argument that visual-to-neural mappings are inherently one-to-many? Could the authors clarify whether this deterministic mapping risks averaging out noise or inter-trial variability, and what impact this has on downstream synthesis quality and biological realism?\n\nLIMITATIONS:\nSee weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes SynBrain, a generative framework for visual-to-fMRI synthesis that models neural responses as semantic-conditioned probability distributions. The key innovation lies in addressing the inherent one-to-many mapping from visual stimuli to neural responses through probabilistic modeling. SynBrain consists of two main components: (1) BrainVAE, a variational autoencoder that models fMRI signals as continuous probability distributions conditioned on visual semantics, and (2) S2N (Semantic-to-Neural) Mapper, a Transformer-based module that maps CLIP visual embeddings to the fMRI latent space. The authors evaluate their approach on the Natural Scenes Dataset (NSD) with 4 subjects, demonstrating superior performance in subject-specific visual-to-fMRI synthesis and effective few-shot adaptation to novel subjects.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\n**Quality:**\n- The paper addresses a fundamental challenge in computational neuroscience by explicitly modeling the one-to-many nature of visual-to-neural mappings, which is biologically motivated and theoretically sound.\n- The experimental design is comprehensive, evaluating across voxel-level, semantic-level, and retrieval metrics, providing multi-faceted validation of the approach.\n- The ablation studies clearly demonstrate the contribution of each component (variational sampling, contrastive learning, S2N Mapper).\n- The architecture design is well-motivated, particularly the use of convolutional and attention layers in BrainVAE to address instability issues in MLP-based VAEs.\n\n**Clarity:**\n- The paper is generally well-written with clear motivation and methodology sections.\n- The figures effectively illustrate the approach and results, particularly Figure 2 showing the overall framework.\n- The experimental setup is clearly described with sufficient implementation details for reproducibility.\n\n**Significance:**\n- The work addresses an important limitation in existing deterministic approaches to brain encoding, offering a more biologically plausible framework.\n- The results demonstrate practical benefits for data augmentation in few-shot fMRI-to-image decoding scenarios.\n- The findings about cross-trial and cross-subject functional consistency provide insights into neural variability patterns.\n\n**Originality:**\n- The combination of probabilistic modeling with semantic conditioning for fMRI synthesis is novel and well-motivated.\n- The S2N Mapper design offers a more efficient alternative to diffusion-based approaches used in prior work.\n- The framework successfully balances modeling neural variability while preserving functional consistency.\n\n**Weaknesses:**\n\n**Quality:**\n- The evaluation is limited to a single dataset (NSD). While this is the largest and highest quality dataset available, validation on additional datasets would strengthen the claims, and I am particulary interested to see how the variational approach proposed would handle lower quality and smaller datasets.\n- The comparison baseline is primarily limited to MindSimulator. The paper would benefit from comparisons to other established image-to-fMRI encoding models, such as the GNet model introduced in \"Brain-optimized deep neural network models of human visual areas learn non-hierarchical representations\" (St-Yves et al., 2023) and the Transformer-based approach from \"Predicting Human Brain States with Transformer\" (Sun et al., 2024). These comparisons would better contextualize SynBrain's performance relative to the broader landscape of visual encoding models, even if these baselines are deterministic.\n\n**Clarity:**\n- Some technical details about the BrainVAE architecture could be clearer, particularly the specific design choices and architectural details for the convolutional and attention components. This could have been bolstered by making code accessible to reviewers as well.\n\n**Significance:**\n- While the results are promising, the practical impact may be limited by the computational overhead of the probabilistic approach compared to deterministic methods. Can the authors comment on the computational efficiency of the method compared to existing approaches?\n- The few-shot adaptation results, while encouraging, are evaluated on relatively similar subjects from the same dataset and the same scanner. Without cross-dataset adaptation results it is difficult to quantify the true impact of this result.\n\n**Originality:**\n- While the overall approach is novel, individual components (VAEs, contrastive learning, Transformers) are well-established techniques.\n- The novelty lies primarily in their combination and application to fMRI synthesis rather than fundamental algorithmic innovations.\n\nQUESTIONS:\n1. How does the computational cost of SynBrain compare to deterministic baselines? The paper mentions faster training but doesn't provide detailed timing comparisons or discussions. Is this impacted by the necessary computing hardware to train and run their method? The authors note that they use 4 A100's to train their method, it is possible to train it on a single GPU or with reduced hardware requirements? Does  the variational approach have meaningful differences in inference efficiency?\n\n2. The authors claim that synthesized fMRI signals achieve higher retrieval accuracy (92.5%) than raw fMRI (84.8%). Could this indicate that the model is learning to enhance or denoise the signals rather than faithfully reproduce the noisy properties of neural responses? It seems to me that one of the primary advantages of a variational system is the ability to more accurately model the noise properties of real neural data, what do the authors make of this point in the context of the \"better\" retrieval scores?\n\n3. How sensitive is the approach to the choice of visual encoder (CLIP)? Have the authors experimented with other pre-trained vision models?\n\n4. The few-shot adaptation experiments use subjects from the same dataset. How would the approach perform when adapting to subjects scanned with different protocols or equipment? Transferring to new subjects with minimal data in new datasets would be of significant interest to downstream applications of the approach and would boot the significance of the work.\n\n5. Could the authors provide more analysis on what specific types of neural variability the model captures versus what it might be missing?\n\n6. On line 171 the authors note that MindEye uses a diffusion transformer, while the MindEye paper actually uses a *diffusion prior*. This detail should be corrected for the camera ready version.\n\nLIMITATIONS:\nThe authors adequately address limitations in the conclusion, including reliance on pre-trained vision models and incomplete modeling of variance sources. However, they could expand on computational requirements, unknown dataset transferability, and any scalability concerns.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces SynBrain, a generative framework that operates under the brain decoding scenario. Experiments show that SynBrain generates more coherent data compared to the baseline and can improve existing decoding models as a data augmentation engine under limited-data scenarios.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths\n- The proposed data synthesis pipeline outperforms the shown baselines.\n- The proposed method can also be utilized to improve existing decoding models as a data augmentation strategy under limited-data scenarios.\n\n## Weaknesses\n- There are some concerns regarding the theoretical motivation of the proposed loss function (Eq. 4); more on this in the questions section.\n- Details on some of the experiments, mainly figure 3, seem to be absent.\n\nQUESTIONS:\n- What kind of fMRI input was used? The \"nsdgeneral\" region, like previous works such as MindEye\n- For the data augmentation experiments in table 2, did the authors try traditional data augmentation strategies as a comparison?\n- Experimental details for figure 3: \n    - Are the experiments about just swapping the brain encoder/decoder from the pipeline in figure 2?\n    - Is the MSE loss evaluating only the autoencoder, while the other losses evaluate the synthesis pipeline?\n- I'm not sure if it's a good idea to introduce two (possibly competing) objectives that directly optimize the latent z in equation 4. Is it really necessary to introduce the SoftCLIP loss to the latent, instead of some other part of the pipeline, when it is already conditioned by the KL loss? I understand that outright removing the SoftCLIP loss is harmful, as seen in table 3, but can't it be introduced in some other part instead of being directly fed into the latent z that's supposed to follow a Gaussian distribution?\n- On a related note, how do the KL loss and SoftCLIP loss behave during training? Do $\\lambda_{KL}$ and $\\lambda_{CLIP}$ keep the two losses in the general ballpark during training? Do the latent vectors of the training samples properly follow a Gaussian distribution after training?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis current model addresses a one-to-many relationship in stimulus to brain mapping by transforming semantically rich visual stimuli into synthesized neural activity patterns. The authors introduce a non-deterministic model in which both seen and unseen visual stimuli are used to generate synthesized fMRI data. The model performance is reported through a series of comparisons between raw and synthetic voxel activity as well as the resulting generated images' similarity to the original images.\n\nSTRENGTHS AND WEAKNESSES:\n### Pros:\n\n* The current model addresses high variability across humans (and thus across differences in individual visual systems), accounting for variance across physiological and trial-level constraints.\n\n* They employ a novel non-linear transformation of CLIP embeddings into fMRI latent space.\n\n* All together, the model is built to generate fMRI signals that represent high-level visual processing regions.\n\n### Cons:\n\n* (Very, very minor comment) please correct the typo in Figure 3 (Image Retrieval Accuracy).\n\n* The dataset on which the model was tested had very few subjects. The Natural Scenes Dataset only contains 8 subjects in total (albeit several hours of viewing per subject), of which this paper could only utilize 4 subjects due to experiment completion. Please consider a simulation testing variability across more simulated individuals.\n\n* Visual images generated (Figure 4) seem to capture the gist of the input visual stimulus but have some strange distortions.\n\nQUESTIONS:\nHave you considered using a simulation to generate a dataset with more subjects? It is a bit hard to state that the model tested variability across humans (rather than within individuals) with only 4 subjects. \n\nfMRI-to-Image vis -- some of the images generated from the synthesized fMRI capture the gist of the input visual stimulus but have very odd non-sensical distortions to them (e.g. the teddy bear has too many limbs; background vehicles in bus image), how would you explain these distortions? The model is supposed to capture high-level visual representations but seems like it is not reproducing semantic features in the background (vs the foreground).\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}