MindSimulator: Exploring Brain Concept Localization via Synthetic fMRI
Authors: Guangyin Bao
Qi Zhang
Zixuan Gong
Zhuojia Wu
Duoqian Miao
Venue: ICLR 2025 Poster (2025)
Keywords: Neuroscience
fMRI encoding
Generative model
fMRI generation
fMRI functional localizer
Concept-selective voxel
Primary area: applications to neuroscience & cognitive science

Abstract: Concept-selective regions within the human cerebral cortex exhibit significant activation in response to specific visual stimuli associated with particular concepts. Precisely localizing these regions stands as a crucial long-term goal in neuroscience to grasp essential brain functions and mechanisms. Conventional experiment-driven approaches hinge on manually constructed visual stimulus collections and corresponding brain activity recordings, constraining the support and coverage of concept localization. Additionally, these stimuli often consist of concept objects in unnatural contexts and are potentially biased by subjective preferences, thus prompting concerns about the validity and generalizability of the identified regions. To address these limitations, we propose a data-driven exploration approach. By synthesizing extensive brain activity recordings, we statistically localize various concept-selective regions. Our proposed MindSimulator leverages advanced generative technologies to learn the probability distribution of brain activity conditioned on concept-oriented visual stimuli. This enables the creation of simulated brain recordings that reflect real neural response patterns. Using the synthetic recordings, we successfully localize several well-studied concept-selective regions and validate them against empirical findings, achieving promising prediction accuracy. The feasibility opens avenues for exploring novel concept-selective regions and provides prior hypotheses for future neuroscience research.

Published as a conference paper at ICLR 2025
MINDSIMULATOR: EXPLORING BRAIN CONCEPT LOCALIZATION VIA SYNTHETIC FMRI
Guangyin Bao', Qi Zhhan'g, Zixuan Gong', Zhuojia Wu', Duoqian Miao 'Tongii University
ABSTRACT
Concept-selective regions within the human cerebral cortex exhibit significant activation in response specific visual stimuli associated with particular con- cepts. Precisely localizing these regions stands as crucial long-term goal neuroscience grasp essential brain functions and mechanisms. Conventional experiment-driven approaches hinge on manually constructed visual stimulus col- lections and corresponding brain activity recordings, constraining the support and coverage of concept localization. Additionally, these stimuli often consist of con- cept objects in unnatural contexts and are potentially biased by subjective pref- erences, thus prompting concerns about the validity and generalizability of the identified regions. To address these limitations, we propose a data-driven ex- ploration approach. By synthesizing extensive brain activity recordings, we sta- tistically localize various
regions. Our proposed MindSimula- for leverages advanced generative technologies to learn the probability distribu- tion of brain activity conditioned on concept-oriented visual stimuli. This en- ables the creation of simulated brain recordings that reflect real neural response patterns. Using the synthetic recordings, we successfully localize several well- studied concept-selective regions and validate them against empirical findings, achieving promising prediction accuracy. The feasibility opens avenues for ex- ploring novel concept-selective regions and provides prior hypotheses for future neuroscience research.
INTRODUCTION
The human brain's visual cortex is decisive in processing and perceiving visual information. Neu- roscience researchers have long dedicated themselves to unraveling the brain's visual mechanisms. making impressive strides such as in brain visual encoding (Mitchell et al. 2008). decoding (Gong et al. 2024b), and visual perception (Chen et al. 2020 However, the process of forming visual cognition remains to be explored. Notably, localizing the various functional organizations and ac- tivation patterns of the visual cortex that correspond to human conceptual cognition is considered pivotal yet remains challenging frontier (Huth al., 2016) [enderson al., 2023 Luo et al.) 2024]. Numerous neuroscience studies have illustrated that specific regions of the visual cortex ex- hibit concept selectivity. When individuals receive visual stimuli related particular concepts (such as places, bodies, faces, words, colors, and foods), the respective cortical regions exhibit significant activation (Epstein & Kanwisher 1998 Sergent et al. 1992 Jain et al. 2023 Pennock et al.) 2023] Kanwisher et al.) [197 et al. 2202). These regions are termed visual concept-selective regions and play a vital role in advancing the understanding of brain visual cognition.
Typically, identifying concept-selective regions relies the functional localizer (flooc experi- ments (Stigliani et al.) 20155] Allen et al.] (022]. To this end, neuroscience researchers need to purposefully and manually construct visual stimulus sets associated with specific visual concepts. These stimuli are subsequently presented to subjects for costly functional magnetic resonance imag- ing (fMRI) scans, aiming to localize these regions by statistically analyzing the fMRI data related to visual stimuli. However, this experiment-drivee exploration encounters three major limitations: 1) Real fMRI-image data scarce, resulting in colcept-selective region localization being limited
"Corresponding author.

---

Published as a conference paper at ICLR 2025
to few concept categories. 2) The collection of visual stimuli accompanied by an artificial selec- tion is biased. 3) Existing manual construce visual stimuli sets often consist of isolated objects in unnatural scenes. These limitations naturally prompt concerns about the generalizability of vi- sual concepts of concept localization. To overcome these issues, we intend to leverage a flexible data-driven approach to break through the limitations of manual-constructed stimuli and expensive experimental IMRI collection to locate more generalized and precise concept-selective regions.
In this paper, we propose MindSimulator, a novel generative fMRI encoding model for flexibly syn- thesizing individual fMRI corresponding to concept-oriented visual stimuli. MindSimulator operates through a reverse process in conjunction with fMRI decoding (Naselaris al. 2011). Building on the significant advancements in fMRI visual decoding (Scotti et al. 22224 by Gong et al. 2024a]
Shen et al. 2024), we are motivated to develop powerful fMRI encoding models by leveraging ex- isting fMRI datasets (Allen et al. 2022) and advanced generative deep learning techniques (Ho et al., 2020 Rombach et al. 2022). Specifically, MindSimulator first constructs an fMRI autoencoder and aligns [MRI latent space with well-trained visual stimuli (i.e., image) representation space. Sub- sequently, a diffusion model is integrated to learn fMRI's conditional probability distribution for a given concept-orienter visual stimuli on the fMRI-image joint representation space. Once Mind- Simulator is trained effectively, it serves as an individual's 's brain capable of generating fMRI data corresponding to diverse concept-related visual stimuli ideally.
In addition, we evaluate the fidelity of synthetic fMRI at the voxel level and semantic level, ensuring that MindSimulator tic contained in IMRL. More importantly, MindSimulator experimentally shows excellent generaliza- tion capability, even for out-of-distribution visual stimuli, thereby enabling synthesizing extensive IMRI of various concepts and achieving an expansion for scarce fMRI data.
maximally restore recognizable neural response patterns, i.e., visual seman-
On this basis, we use synthetic fMRI localize soncept-selective regions. Statistically, the data- driven localization enables us to explore concept-selective regions across various categories, facili- tating more finer-grained region discovery, instead of being limited to the categories present in the fLoc experiment's ' stimuli categories. We conduct concept-selectivg region localization experiments using fMRI synthesized by Mindšimulator and verify its feasibility by predicting the existing re- gions empirically delineated by the fLoc experiments.
2
RELATED WORKS
FMRI Encoding. The fMRI encoding research has been explored over long period (Mitchell et al., 2008 Huth al.) 2016: Gu et al., 2022, Tang et al. 2023). Existing approaches used regression models to map image features to voxel space. Some researchers focused selecting better image features (Han et al. 2019 Wang et al. 2023) or better visual stimuli (Luo et al. 2024), and simple used for better interpretability. The others were concerned with developing better regression models (Gifford et al. 2023) Yang et al., 2023] Adeli et al.) 2023 Ma et al. 2024
linear regression
Liang et al. 2024 Beliy et 2024). We are the first to develop generative encoding model.
Generative Models. Generative models can sample from noise to generate data with clear se- mantics. Mainstream generative architectures include variational autoencoders (Kingma 2013 Van Den Oord et al.] 2017), generative adversarial networks (Goodfellow et al.] 2014), and diffusion models (Sohl-Dicksteee
et al. 2015 Ho al. 20200 Song al. 2020). Generative models condi- tional on images can output text (Li el al.) 22222 (022,2023 2023). immags (Xu et al. 2023 Zhang et al. 2023
or videos (Blattmann al.
2023 Shi et al. (024).
FMRI Visual Semantic Decoding. Advanced brain decoding studies have been able to decipher clear visual semantics from brain activity fMRI recordings and restore seen visual scenes (Lin al.
2022 Chen et al.) 2023) Scotti et al., 2024a by Wang et al. 2024; Quan et al.) 2024] Gong et al., 2024a Shen et al.] 2024 [ao et al. [024 Xia et al. 2024 Chen al. [024). demonstrating their
strong capability for recognizing global neural response patterns. In our study, these pre-trained
visual decoding models are components of our semantic-level evaluation pipeline.
METHOD
Our MindSimulator aims to map arbitrary visual stimuli into individual brain activity using paired natural image stimuli and fMRI recordings. We begin by articulating our motivation for adopting the
2

---

Published as a conference paper at ICLR 2025
generative architecture and provide general overview of the proposed method. We then describe each key component MindSimulator. Finally, we describe how noise can be translated into cortical activity fMRI recordings conditioned on given visual stimuli.
fMRI Autoencoder
Regression fIARI
Diffusion Estimator
Nosse
spece
Myph
JW
MSE
Voxel Decoder
SeRCLIP
Image Estractir
Diffision Transfemer
Visual Inference Sampler
Enibedding
Image Extractor
Visce Encader
Denoising
Synthesic 1MR)
MW SenU
Deflusion
Voxel Decader
MM
Transformer
Visual Stimall
Figure 1: Overview of the proposed MindSimulator. comprises IMRI autoencoder, Diffusion Estimator, and Inference Sampler. The IMRI autoencoder enables mutual transformation between voxels and IMRI representations. The diffusion estimator generates fMRI from noise conditioned on images. The inference sampler achieves high-precision fMRI synthesis. Please refer to Scctions 3.2 to 3.4 for more details.
3.1 MOTIVATION AND OVERVIEW
At least two types of approaches can be used for mapping visual stimuli their representations to corresponding fMRI recordings. The regression models directly map visual stimuli to target fMRI recordings through deep network; whereas the generative models visual stimuli conditional guidance to generate the target fMRI recordings from Gaussian noise. However, observations show that there are noticeable differences in brain activity IMRI recordings even when receiving the same visual stimuli (Horikawa & Kamitani, 2017 Allen et al., 2022), suggesting that visual stimuli and fMRI recordings essentially exhibit a monyto-many correspondence. Regression models fail cap- ture this phenomenon because the mapped fMRI recording of a given visual stimuli is unique; on the contrary, generative models treat differential fMRI recordings as outcomes of random sampling from conditional probability distribution based on given visual stimuli. Therefore, the mechanism of generative models aligns more closely with the behavioral performance of the human brain.
The proposed MindSimulator adopts generative architecture. As illustrated in Figure I it con- sists of three components: 1) a fMRI Autoencoder, which facilitates the interconversion between IMRI voxel and its high-dimensional representation; 2) Diffusion Estimator, which learns the conditional distribution of fMRI based on given visual stimuli; and 3) a Inference Sampler, which generates accurate synthetic fMRI using multi-trit trial enhancement and correlated noise.
3.2 FMRI AUTOENCODER
The low signal-to-noise fMRI involves complex brain activity, bringing difficulty to estimating its data distribution. Thus, we project raw fMRI into nigh-dimensional latent representation via the fMRI autoencoder and estimate its conditional distribution in the latent space instead. Specifically, sampling a paired training data (x. from subject-individual fMRI dataset S, where x E R de- notes preprocessed fMRI blood oxygenation level-dependent (BOLD) voxels and denotes the cor- responding visual stimuli. The autoencoder consists of voxel encoder E(-) and voxel decoder D(-). The voxel encoder embeds resulting in m d-dimensional tokens. The voxel decoder works just the opposite, decoding the high- dimensional voxel representation A' back to fMRI voxels, i.e. å D(X) R Finally, we train this autoencoder using strong voxel-wise supervision objective: Cmse = Ez-s - L|| To simplify, omit the notation related the mini-batch sampling.
nigh-dimensional fMRI representation X' = E(x) Rm
Inspired by existing text-to-image generative models (Rombach et al. 2022 Saharia et al. 2022)
further construct modsl joint latent spaces to facilitate stable convergence of generative models. Therefore, we align the fMRI representation space with a pre-train image representation space. Specifically, trained CLIP ViT' (Radford et al. 2021) the image extractor V--))

---

Published as a conference paper at ICLR 2025
for visual stimuli. Note that the previous study has shown that image representations obtained by contrastive learning are more suitable for fMRI encoding task (Wang et al. 2023). The image repre- sentation y V(y) E R has consistent dimension with fMRI representation X. Subsequently, we use SoftCIIP loss (Gioo et al. 2024) with a cosine-scheduled temperature factor r to supervise
the cross-modal alignment process:
[S] [8]
1
exp(As x2//T)
exp(Xi. -2//T)
Sofftllip
-log
[
exp(X; Nx/T)
| k= exp()/ (K/T))
(1)
[S]
[s]
1
exp(Y's ///T)
exp(X; V//TT)
loo
S
Σ k=1 (): Dx/T)
Si k= exp(Xi 2 D2/T))
We fix the pre-trained visual extractor and train the fMRI autoencoder end-to-end with joint loss:
Lautoencoder
- + Lsoftclip
3.3 DIFFUSION ESTIMATOR
Contrastive learning facilitates disjointed cross-modal representations (Ramesh al. 2022). Ac- cordingly, the IMRI representation X' output by the voxel encoder is parallel with the corresponding image representation while keeping a certain distance. On this basis, we train generative model to learn the conditional probability distribution of fMRI representation on a given image represen- tation. Previous study (Oko et al. 2023) has shown that diffusion models are suitable to achieve the goal. Accordingly, we construct the diffusion estimator P(.) with 7 timesteps. In the estimator, by applying the reparameterization trick, the noised fMRI representation 2, can be formalized as:
z = Vos X
αe ag = II am: t ~~ [1,2, ... ,T,,
(3)
where denotes the noise schedule hyperparameter and e ~ A( is Gaussian noise. Unlike common diffusion models that predict noise (Ho et al.) (020). we aim to learn the conditional dis- tribution such that our diffusion estimator directly predicts target fMRI representations. Using T, to denote learnable time embedding of timestep f, learning objective can formalized as:
Laiffusion E.., P(Z V. - -
Here, adopt Transformer architecture (Vaswani al.
2017 Peebles & Xie 2023) for diffusion
estimator P()), integrating image representations as conditions through cross-attention modules.
3.4 INFERENCE SAMPLER
Once the diffusion estimator is trained, it can progressively predict target fMRI representation A conditional the given image representation, formalized as:
2 P 2 ~ N(O, 1), 8=
(5)
To acquire more accurate synthetic brain activity fMRI, we introduce the following two strategies to improve synthesis performance.
Multi-Trial Enhancement. In neuroscience, intra-subject voxel-wise reproducibility is crucial for experimental exploration. Specifically, it involves comrllating fMRI data across multiple trials of the same visual stimuli to ensure the precision neural responses. Inspired by this treatment, we consider the reproducibility of multiple generated IMRL. We generate N fMRI from N different Gaussian noise, simulating the viewing an image N times. These synthetic IMRI correspond to the same visual stimuli. Finally, average synthetic fMRI to achieve
accurate generation.
Correlated Gaussian Noise. We target synthetic fMRI to exhibit a high correlation in neural re- sponse patterns across the N trials, as lower variance typically leads to increased fMRI synthesis performance. It is well known that the uncertainty in the generated results arises from the random- ness of the Gaussian noise; thus, propose to correlated Gaussian noise the input for N-trial
4

---

Published as a conference paper at ICLR 2025
generation. To create N correlated Gaussian noise, first randomly sample two independent noise 61 ~ (0, and E2 ~ NO00 1). Then, we apply weights and thereby obtain N new noise en²
c, vBn é + -1- B, E2. [1, .N]
(6)
By setting different weights 3,. we can obtain series of correlated standard Gaussian noises, which are located on the curve between 61 and 62 in the high-dimensional space and have high similarity.
EXPERIMENTS SETUP
4.1 DATASETS
We use the Natural Scenes Dataset (NSD) (Allen et al.) 2022). which is an extensive whole-brain fMRI dataset gathered from 8 subjects viewing images from MSCOCO (Lin et al. 2014). In NSD experiments, participants were required to view 10,000 images for 3 trials, thereby acquiring 30,000 IMRI scans. Our evaluation focuses on Subj01, Subj02, Subj05, and Subj07 because these subjects completed all experiment sessions. The ~9,000 unique images for each subject are used for training and the remaining ~1,000 shared images are used for evaluation. During the training phase, all three fMRI of the same image are used individually; while for testing, three repeats are averaged. We use beta-activations computed using GLMSingle (Prince et al. (022) and normalize each voxel to j = 0, 0 = 1 on per-session basis. We average multi-trail voxels for the testing set. Since we are focusing on the visual cortex regions, we apply the official nsdgeneral region-of-interes (ROI) mask, which spans visual regions ranging from the early visual cortex to higher visual areas. We flatten the fMRI selected by ROI and then obtain one mie-dimenioonal voxel sequences for encoding.
4.2 IMPLEMENTATION DETAILS
For the image extractor, used pre-trained CLIP ViT-L/14. It encodes image representation with the dimension of 257x768. Our voxel encoder consists sequentially of MLPs and residual networks while the voxel decoder is just the opposite. We trained the fMRI autoencoder end-to-end for 300 epochs, using AdamW (Loshchiloy) 2017) with cycle learning rate schedule starting from 3e-4. For the diffusion estimator, we set the timesteps T 100, adopting a cosine noise schedule and 0.2 conditions drop. The diffusion network contained 6 Transformer blocks. Each block computes at- tention using 257 image tokens, 257 noised fMRI tokens, and I time embedding. We train for 150 epochs using gradient clipping, with the same learning rate our autoencoder. For hyperparameter 9, we randomly sample from U10 1). All components of our MinuSimulator can be trained using single NVIDIA Tesla V100 GPU. For a single subject, the training is about 12 GPU hours for the fMRI autoencoder and 20 GPU hours for the diffusion estimator. During the inference phase, takes only 300ms to synthesize an IMRL. Refer to Appendix A.1 for more implementation details.
4.3 EVALUATION METRICS
How do we evaluate the performance of fMRI encoding models? Previous studies (Gu et al. 2022; Wang et al. 2023 Luo et al., 2024) used vexel-level metrics, such as Pearson correlation, voxel-wise mean square error (MSE), and R-squared. However, local-focused voxel-level met- rics are limited because they overlook global accuracy. Specifically, these metrics fail to evaluate whether syn- thetic fMRI accurately preserves the original neural re- sponse patterns, which are the root of human visual phe- nomena. We utilize images for an analogous explanation. As shown in Figure 2 we generate two predictions for ground truth image. It can be seen that Prediction has better pixel-level accuracy (pixel-wise MSE) but lacks recognizable global pixel patterns (image semantics), whereas Prediction 2 shows the opposite. Consequently, we might conclude that Prediction 2 is superior. Similarly, the same issues would arise with fMRI synthesis, highlighting the need to introduce semantic-level metrics.
MSF-0.055385
MSE=D. 173868 M7860.173868
Prediction
Prediction
Analogical explanation for lim- itation of voxel-level metrics. The better low-level performance does not indicate more accurate synthesis.
Figure 2:
Thanks to advanced research on fMRI visual decoding tasks (Scotti et al. 2024a we can incorpo- rate semantic-level evaluation for synthetic fMRI. The trained 1MRI decoding model can interpret
5

---

Published as a conference paper at ICLR 2025
individual neural response patterns and reconstruct original visual stimuli. Thus, when the synthetic fMRI aligns with the generalization capacity of the trained visual decoding model, if it retains the neural response pattern, it be recognized by fMRI decoding models and the reconstructed visual stimuli should be similar to the seen visual stimuli. Based on the above reflections, we propose the semantic-level evaluation pipeline for synthetic fMRI, shown in Figure 3 We utilize decoding model to reconstruct visual stimuli based on synthetic IMRI and then compare it with ground truth stimuli using image reconstruction metrics. Among these metries, PixCorr, SSIM, Alex(2), and Alex(S) are used to evaluate whether the synthetic fMRI retains perceptual relevann neural patterns, while Incep, CLIP, Eff, and SwAV are used to evaluate concept-relevan. neural patterns. As for the decoding model, we use the trained MindEye2 high-level model (Scotti al. 2024b). More details on semantic-leved metrics can be found in Appendix A.2
Synthoti fMR
Enceding Model
AANiAa WANAWAA MAN MaMaaa
Trained Deccding Model
Ground Truth Stimali
Reconstrucied Stimuli
Figure 3: The proposed semantic-level evaluation pipeline for synthetic IMRL. We trained visual decoding models extract the semantics contained the synthetic IMRI and compare them with the ground truth.
5 RESULTS
5.1 EVALUATION FOR SYNTHETIC FMRI
Generating accurate fMRI is crucial for localizing concept-selective regions. We evaluate the encod- ing accuracy of synthesized fMRI using the voxel-level and semantic-level metrics. For comparison, select two representative encoding models as baselines: the linear regression model, known for its strong interpretability with wide applications in neuroscience research (Huth et al., 2016 Wang al. 2023 Tang et al. 2023 Luo al. 2024), and the Transformer encoding model (Adeli al.
2023 which has performed exceptionally ly well in a fMRI encoding challenge (Gifford et al. 2023)
and is increasingly used in recent studies (Liang et al.) 2024; [Beliy et al., 2024). Details on the implementation and training for the baseline encoding models can be found in Appendix A.3 We
also compare the semantic-level metrics calculated using ground truth fMRI, serving as an upper bound of the encoding performance. The quantitative evaluation results are shown in Table [
Overall, our MindSimulator outperforms baselines in encoding accuracy. Additionally, its perfor- mance approaches the upper bound, suggesting minor differences in voxel-wise similarity and global neural response patterns between synthetic IMRI and ground truth fMRI. We further validated the above conclusion through visualization results. As shown in Figure 4 the fMRI synthesized by MindSimulator is significantly better than that of the linear regression model, indicating it contains more accurate neural response patterns, and thereby models. Moreover, compared with seen visual stimuli, the differences in semantics are almost neg- ligible. Such results encouraging because they sufficiently accurate and thereby allow to explore neuroscience findings using synthetic iMRI instead of scarce ground truth fMRI.
be recognized by trained fMRI decoding
In addition, two facts need be further discussed. One is the decoded results from the synthe- sized IMRI have noise, which we attribute to minor voxel-wise differences. Since the decoding model trained using ground truth fMRI, they struggle to accurately decode the mildly distorted synthetic fMRI. The other is that when we apply the multi-trial enhancement strategy, both MSE and semantic-level metrics improve simultaneously. This out of our expectation, as increased voxel-wise reproducibility typically dilutes the accuracy of global neural response patterns. We at- tribute the observed fact our correlated noise, which reduces variance in the neural pattern across multiple fMRI encoding trials, allowing multi-trial averaging to enhance semantic-level metrics.
5.2 DUT-OFFDIISTRIUUTIO GENERALIZATION
Further evaluation of the generalization performance of MindStimulator is essential. Although has shown promising encoding performance MSCOCO, neuroscience researchers may require its ap-

---

Published as a conference paper at ICLR 2025
Voxel-Level
Semantic-Lowe Alex(5)f
Method
Pearson?
MSE
PixCorrt
SSIMF
Alex(2y7
Incop?
CLIPT
Emt
SwAV]
GT FMRI (upper bound)
0.278
0.328
95.2%
99.0%
96.4%
94.5%
0.622
0.343
Linear Regressive
0.334
0.394
0.174
0.266
85,4%
94,2%
90.1%
87.25
0.728
0.432
Transformer Encoding
0.337
0.387
0.166
0.286
83.5%
93.0%.
85.5%
0.759
MindSimulator (Trials=1)
0.345
0.404
0.194
0.296
89.0%
96.2%
92.3%
90.3%
0.702
0.399
MindSimulator (Trials=5)
0.355
0.385
0.201
0.298
89.6%
96.8%
93.2%
91.2%
0.688
0.393
Table I: Evaluation Results of IMRI synthesis accuracy. report average values the 4 subjects. Our MindSimulator achieves optimal performance in both voxel- metrics and semantic-level metrics.
GT
Linesr
Ours (maal11)
Ours (Trial
Figure 4: Visualization comparison between linear regression encoding and our MindStimulator. GT seen visual stimuli. Linear reconstruction from linear encoded fMRL. Ours reconstruction from encoding. The IMRI synthesized by our method has more accurate concepts, colors, backgrounds, and number of objects. We show the results of Subj01 and more results can be found in Appendix B Zoom in for better viewing.
plication on other image datasets. Thus, its ability to encode images from different datasets is equally important. To evaluate this, we utilized CIFAR-10 and CIFAR-I00 (Krizhevsky et al.) 2209), which have distinct image distributions compared to MSCOCO. We select all 10,000 images from their test sets for encoding. Due to the lack of ground truth IMRI, we only compute the semantic-level metrics. The quantitative results are presented in Table 2 while qualitative visualizations are shown in Figure [ We can see that encoding with CIFAR- 10/ 100 images demonstrates only a minimal decrease in most metrics, and in some metrics, even shows improvement compared to MSCOCO. Additionally, the visualization results also indicate that the synthetic fMRI still retains accurate vi- sual semantics. The results demonstrate that MindStimulator has nut-of-distribution generalization capability, suggesting we can use a wider range of image data for neuroscience exploration.
Semantic- Leve Alex(5y]
Datasets
PixCorr
SSIM?
Alex(2)1
Incep"
CLIP
Eff.
SwAVI
MSCOCO
0.201
0.302
89.5%
96.8%
91.5%
88.7%
0.724
0.409
CIFAR-10
0.269
0.406
88.5%
94.3%
84.3%
90.5%
0.898
0.645
CIFAR-100
0.260
0.420
86.6%
93.0%
82.8%
86.3%
0.916
0.659
Table 2: Evaluation of out-of-distribution generalization. We report the results of Subj01. Our MindSimulator demonstrates excellent fMRI synthesis performance on out-of-distribution datasets.
5.3 ABLATION
We conduct ablation experiments to investigate the necessity of each design in MindSimulator. First, we ablate the Voxel Decoder, aligming outputs of the Voxel Encoder with the image latent space un- der the supervision of SoftCLIIP, while the diffusion estimator directly predicts the voxels instead of fMRI representations. Second, we ablate the fMRI-stimuli joint latent space, training the entire IMRI Autoencoder solely using the MSE loss. Furthermore, we ablated the high-dimensional fMRI representation space by removing the fMRI Autoencoder, which means that the diffusion estimator is trained only in the original voxel space. Finally, we also ablate the correlated noise. The experi- mental results are displayed in Table As a role in improving the fMRI encoding performance. In addition, they also contribute to the stable convergence of our diffusion estimator.
see, each component of our MindSimulator plays
7

---

Published as a conference paper at ICLR 2025
CIFAR-10
Ours Stimuli
Datasets: CIFAR-100
Ours Stimuli
Figure 5: Comparison between C1FAR-10/100 images (Stimuli) and corresponding reconstructing results from MindSimulator's synthetic fMRI (Ours). The original stimuli are upsampling to 224x224.
Voxel-Level
Semantic-Level Alex(5)"
Methods
Pearsonf
MSEL
PixCout
SSIM
Alex(2)
Incept
CLIP
Eff
SwAVI
MindSimulator
0.322
0.418
0.204
0.302
90.9%
96.5%
93.0%
89.8%
0.712
0.407
Voxel Decoder
0.287
0.457
0.196
89.8%
95.8%
87.7%
86.6%
0.755
0.428
-wlo Joint Latent Space
0.215
0.524
0.181
0.293
87.3%
93.5%
85.3%
82.5%
0.791
0.456
-wfo IMRI Autocncoder
0.203
0.843
0.152
0.295
82.8%
89.4 99.4%
78.2%
75.2%
0.855
0.506
-w/ Correlated Noise
0.316
0.431
0,200
0.302
89.8%
96.2%
91.5%
89.6%
0.726
0.404
Table 3: Ablation experiments on each component of MindSimulator. We report the results of Subj01. Each
component of our method is crucial for improving fMRI encoding performance.
LOCALIZING
SEEECPT-EEEEEIVEE REGIONS
Our evaluations have demonstrated that synthetic IMRI exhibits neural response patterns that resem- ble ground truth IMRI. This allows synthesize massive fMRI data for neuroscience research, especially in the localization of concept-selective regions. Our localization exploration involves two steps. First, we select concept-oriented visual stimuli and synthesize the corresponding fMRI. We then apply the same statistical analyses as fLoe (Stigliani et al. 2013) to identify concept-selective regions, validating these results against established empirical findings. In the second step, we predict novel concept-selective regions and confirm these predictions through voxel ablation experiments.
6.1 PREDICT EMPIRICAL REGIONS
The NSD dataset localized places-, bodies-, faces-, and words- selective regions through functional localizer (fLoc) experi- ments. As illustrated in Figure 6 simple observation can re- veal great overlap in the regions associated with fLoc words- selective and fLoc faces-selective, while fLoc place-selective and fLoc bodies-selective do not. Consequently, predict places- and bodies-selective regions using fMRI synthe- sized by our MindSimulator, and evaluate the predictions using empirical findings from the existing NSD fLoc experiments.
overlap
words
faces
places
intend to
bodies
Figure 6: The empirical findings of faces-, bodies-, places-, and words- selective regions in NSD fLoc.
The first step localization involves selecting concept-oriented visual stimuli. To achieve this, utilize the pre-trained CLIP model for zero-shot classification, which compulsory assigns MSCOCO images the target concept categories. After these, we select the top-k images with the highest clas- sification probability as visual stimuli used for further exploration. In the NSD fLoc experiments, hundreds of visual stimuli are used, so we similarly select between 100 and 1000 top-ranking im- ages. Figure 7 presents subset of the top 100 selected images and compares them with stimuli used in fl.oc experiments. Significant distribution differences exist.
We encode the selected images into fMRI voxels using both MindSimulator and the commonly adopted linear regression as encoding models. Based on the synthetic fMRI, we compute voxel-wise statistical significance by performing one-sample t-test and Bonferroni correction. This allows us to identify concept-selectin selective regions by setting a significance threshold. To evaluate the accuracy use Accuracy and Fi-score metrics. comparing the
of data-driven region localization, predicted regions with empirical findings. Additionally, we compute the semantic specificity of the images used for encoding, which reflects the strength of relevance hetween the selected images
8

---

Published as a conference paper at ICLR 2025
Category: places
NSO floc
Ours
Category: bodies
NSO floc
Ours
Figure 7: Comparison between visual stimuli used in NSD fLoc experiments and images we use for localiz- ing concept-selective regions. The fLoc experiments visual stimuli that place targets in unnatural scenes, whereas we use the target concept real seenes. They exhibit huge image distribution differences.
places- Specificity
places-Accf
Places-F11
bodies- Specificity
bodies-Acct
bodies-F111
Images
Linear
Ours
Lineur
Outs
Linear
Ours
Lineur
Ours
Top 100
0.9608
32.0%
52.6%
0.461
0.573
0.9988
45.8%
91.7%
0.570
0.674
Top 200
30.0%
45.6%
0.444
0.9977
42.1%
84.2%
0.551
0.727
300
0.9189
29.2%
41.3%
0.437
0.540
0.9968
40.7%
81.4%
0.543
0.740
Top 500
0.8834
28.5%
37.3%
0.430
0.513
0.9953
39.2%
75.3%
0.532
0.732
Top 1000
0.8084
27.9%
33.5%
0.425
0.483
0.9918
37.9%
64.2%
0.523
0.693
Table Localization evaluation of places- and bodies-selective regions. report the results of Subj01. The results are average values obtained with 3 different random seeds
and the target concept. This specificity is calculated by averaging the classification probabilities of all images. More implementation details on localization can be found in Appendix A.4 and the evaluation results presented in Table
The results show that the localization accuracy using fMRI synthesized by MindSimulator sat- isfactory, particularly for bodies-selective regions, despite notable image distribution differences between the NSD fLoc experiments and our selected images. This highlights the feasibility of our data-driven approach for localizing concept-selective regions. Moreover, our localization for places- and selies-ee regions outperforms the linear regression encoding model in both Accuracy and F1 metrics, which we attribute to MindSimulator can synthesize higher-quality fMRI. We also ob- serve that the number of selected images increases, semantic specificity tends to decrease. In addition, there is positive correlation between localization accuracy and semantic specificity, indi- cating that enhancing accuracy requires a focus encoding images with higher semantic specificity.
6.2 EXPLORING NOVEL REGIONS
After confirming that the synthetic fMRI can effectively localize concept-selective regions, we ex- plore the localization of novel regions. Statistically, we can pinpoint any region associated with given concept we select hundreds of images with high semantic specificity from image datasets like MSCOCO CIFAR-10/100. As pioneering exploration, we focus several concepts of interest, including surfer-, plane- food-, and bed-selective regions. We employ CLIP zero-shot clas- sification identify the top 200 images with the highest semantic specificity from MSCOCO and then synthesize fMRI. Subsequently, we perform statistical tests to evaluate voxel-wise activation significance and conduct region localization using the same thresholds. Figure illustrates part of the selected images and their corresponding localization results.
The localized concept-selective regions are predominantly located in the higher visual cortex, as delineated by NSD. This aligns with prior neuroscience knowledge, which indicates that the lower visual cortex shows selectivity for colors and shapes, while the higher visual cortex selective for specific concepts. In addition, it can be seen that the regions localized by different concepts scattered across the cortex with minimal overlap. This suggests that our approach could be extended create concept atlas of the human brain. We also find that localization often consists of

---

Published as a conference paper at ICLR 2025
Concept: Surfer
Concept: Plane
Concept: Food
Concept:
Figure 8: Localized concept-selective selective regions according to synthetic fMRL The red represents the selected regions and darker colors reflect higher confidence. We show the results for Subj01. The results of other subjects and inter-subject comparisons can be found in Appendix Zoom in for better viewing.
Ablated Original
Figure 9: Results of voxel ablation experiments. We mask the surfer-selective region. More precisely, we set the corresponding 1017 voxels (around 6.5%) to negative values. Original = Reconstruction from synthetic IMRL Ablated = Reconstruction from masked synthetic iMRL. It can be observed that when the concept region masked, the two concepts of "surfer man" and "sea" are dissolved separately.
several disconnected regions, which attribute the coupling of related concepts. For example, the concept "surfer" is often coupled with the concepts of "sea" and "human".
To verify the accuracy of the localized novel concept regions, conduct voxel ablation. We mask the synthetic fMRI based on the localized concept-selective regions and then decode us- comprehensive presentation of ablation results,
ing trained decoding models. To provide take "surfer" for illustration. As shown in Figure 9 when the regions are masked, the reconstructed images lose corresponding concept objects, providing partial validation for our localization.
CONCLUSION
In this paper, we introduce MindSimulator, generative fMRI encoding model that utilizes diffu- sion model to learn the fMRI distribution conditional on given visual stimuli in a high-dimensional fMRI-stimuli joint latent space. By employing multi-trial enhancement for sampling, we synthesize massive fMRI data. Our experiments demonstrate that MindSimulator's synthetic fMRI outperforms existing regressive encoding models across both voxel-level and semantic-level metrics, with strong generalization common image datasets. Leveraging synthetic fMRI, conduct data-driven neuroscience explorations, localizing wide-studied concept-selectivc regions and validating these results against empirical findings. We believe that our approach of utilizing synthetic data to enlarge scarce fMRI datasets and then conduct neuroscience research offers an alternative complement to traditional approaches and provides novel hypothetical priors for future exploration.
ACKNOWLEDGEMENTS
This research is supported by the National Natural Science Foundation of China (No. 62376198), the National Key Research and Development Program China (No. 2022 104700,, Shanghai Baiyulan Pujiang Project (No. 08002360429).
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Hossein Adeli, Sun Minni, and Nikolaus Kriegeskorte. Predicting brain activity using transformers. bioRxiv, pp. 2023-08, 2023.
Emily J Allen, Ghislain St- Yves, Yihan Wu, Jesse L Breedlove, Jacob S Prince, Logan T Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, et al. A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence. Nature neuroscience, 25(1): 16-126, 2022.
Guangyin Bao, Zixuan Gong, Qi Zhang, Jialei Zhou, Wei Fan, Kun Yi, Usman Naseem, Liang Hu, and Duoqian Miao. Wills aligner: A robust multi-subject brain representation learner. arXiv preprint ar arXiv:2404. 2024.
Roman Beliy, Navve Wasserman, Amit Zalcher, and Michal Irani. The wisdom of a crowd of brains: A universal brain encoder. arXiv preprint arXiv:2406. /2t:2299..2/79. 2024.
Andreas Blattmann, Tim Dockhorn, Sumith Kulal, Daniel Mendelevitch, Maciej Kilian, Dominík Lorenz, Yam Levi, Zion English, Vikram Voleti, Adam Letts, et al. Stable video diffusion: Scaling latent video diffusion models to large datasets. arXiv preprint arXiv:2311.15127, 2023.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. Advances in neural information processing systems, 33:9912-9922 2020.
Xing Chen, Feng Wang, Eduardo Fernandez, and Pieter R. Roelfsema. Shape perception via a high- channel-count neuroprosthesis in monkey visual cortex. Science, 370(6521) 2020.
Zijiao Chen, Jiaxin Qing, Tiange Xiang, Wan Lin Yue, and Juan Helen Zhou. Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding. In IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 22710-22720. IEEE, 2023.
Zijiao Chen, Jiaxin Qing. and Juan Helen Zhou Cinematic mindscapes: High-quality video recon- struction from brain activity. Advances in Neural Information Processing Systems, 36, 2024.
Russell Epstein and Nancy Kanwisher. A cortical representation of the local visual environment. Nature, 392(6676):598 601, 1998.
Yuting Gao, Jinfeng Liu, Zihan Xu, Tong Wu, Enwei Zhang, Ke Li. Jie Yang, Wei Liu, and Xing Sun. Softelip: Softer cross-modal alignment makes clip stronger. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 1860-1868, 2024.
Alessandro T Gifford, Benjamin Lahner, Sari Saba-Sadiya, Martina G Vilas, Alex Lascelles, Aude Oliva, Kendrick Kay, Gemma Roig, and Radoslaw M Cichy. The algonauts project 2023 chal- lenge: How the human brain makes sense of natural scenes, arXiv preprint arXiv:2301.0311 2023.
Zixuan Gong, Qi Zhang, Guangyin Bao, Lei Zhu, Ke Liu, Liang Hu, and Duoqian Miao. Mindtuner: Cross-subject visual decoding with visual fingerprint and semantic correction. arXiv preprint arXiv:2404.12630, 2024a.
Zixuan Gong, Qi Zhang, Guangyin Bao, Lei Zhu, Yu Zhang, KE LIU, Liang Hu, and Duoqian Miao. Lite-mind: Towards efficient and robust brain representation learning. In ACM Multimedia 2024, 2024b.
Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial nets. Advances in neural information processing systems, 27, 2014.
Zijin Gu, Keith Jamison, Mert Sabuncu, and Amy Kuceyeski. Personalized visual encoding model construction with small data. Communications Biology, 5(1):1382, 2022.
Kuan Han, Haiguang Wen, Junxing Shi, Kun-Han Lu, Yizhen Zhang, Di Fu, and Zhongming Liu. Variational autoencoder: An unsupervised model for encoding and decoding finri activity in visual cortex. Neurolmage, 198: 125-136, 2019.
11

---

Published as a conference paper at ICLR 2025
Margaret M Henderson, Michael J Tarr, and Leila Wehbe. A texture statistics encoding model reveals hierarchical feature selectivity across human visual cortex. Journal of Neuroscience, 43 (22):41444411,, 2023.
Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Advances in neural information processing systems, 33:6840-6851, 2020.
Tomoyasu Horikawa and Yukiyasu Kamitani. Generic decoding of seen and imagined objects using hierarchical visual features. Nature communications, 8(1):15037, 2017.
Alexander G Huth, Wendy A De Heer, Thomas L Griffiths, Frédéric E Theunissen, and Jack L Gallant. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature, 532 (7600):453-44)) -585-88, 2016.
Nidhi Jain, Aria Wang, Margaret M Henderson, Ruogu Lin, Jacob S Prince, Michael J Tarr, and Leila Wehbe. Selectivity for food human ventral visual cortex. Communications Biology, 6(1): 175, 2023.
Nancy Kanwisher, Josh McDermott, and Marvin M Chun. The fusiform face area: A module in human extrastriate cortex specialized for face perception. The Journal of Neuroscience, 17(11): +302-4311, 1997.
Diederik Kingma. Auto-encoding variational bayes. arXiv preprint arXiv: 2013.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convo- lutional neural networks. Advances in neural information processing systems, 25, 2012.
Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping language-image pre-
training for unified vision-language understanding and generation. In International conference machine learning, pp. 12888-12900. PMLR, 2022.
Junnan Li, Dongxu Li. Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language-image pre-training with frozen image encoders and large language models. In International conference on machine learning, pp. 19730-19742. PMLR, 2023.
Ruixing Liang, Xiangyu Zhang, Qiong Li, Lai Wei, Hexin Liu, Avisha Kumar, Kelley Kempski Leadingham, Joshua Punnoose, Leibny Paola Garcia, and Amir Manbachi. Unidirectional brain- computer interface: Artificial neural network encoding natural images to fmri response in the visual cortex. In ICASSP 2024-2024 IEEE International Conference Acoustics, Speech and Signal Processing (ICASSP), pp. 1851-1855. IEEE, 2024.
Sikun Lin, Thomas Sprague, and Ambuj K Singh. Mind reader: Reconstructing complex images from brain activities. Advances in Neural Information Processing Systems, 35:29624-2966 2022.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft COCO: common objects in context. In European Conference on Computer Vision, volume 8693, pp. 740-755, 2014.
I Loshchilov. Decoupled weight decay regularization. arXiv preprint drX1v:1711.05101, 2017.
Andrew Luo, Maggie Henderson, Leila Wehbe, and Michael Tarr. Brain diffusion for visual explo- ration: Cortical discovery using large scale generative models. Advances in Neural Information Processing Systems, 36, 2024.
Shuxiao Ma, Linyuan Wang, Senbao Hou, Chi Zhang, and Bin Yan. Large-scale parameters frame- work with large convolutional kernel for encoding visual fmri activity information. Cerebral Cortex, 34(7), 2024.
Tom M Mitchell, Svetlana V Shinkareva, Andrew Carlson, Kai-Min Chang, Vicente L Malave, Robert A Mason, and Marcel Adam Just. Predicting human brain activity associated with the meanings of nouns. science, 320(5880): 191--1199 1195, 2008
12

---

Published as a conference paper at ICLR 2025
Thomas Naselaris, Kendrick N Kay, Shinji Nishimoto, and Jack L Gallant. Encoding and decoding in fmri. Neuroimage, 56(2):400 410, 2011.
Kazusato Oko, Shunta Akiyama, and Taiji Suzuki. Diffusion models are minimax optimal distri- bution estimators. In International Conference Machine Learning, pp. 26517-26582. PMLR,
2023.
William Peebles and Saining Xie. Scalable diffusion models with transformers. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 4195-4205, 2023.
lan ML Pennock, Chris Racey, Emily J Allen, Yihan Wu, Thomas Naselaris, Kendrick N Kay, Anna Franklin, and Jenny M Bosten, Color-biased regions in the ventral visual pathway are food selective. Current Biology, 33(1):134-146, 2023.
Jacob Prince, Ian Charest, Jan W Kurzawski, John A Pyles, Michael J Tarr, and Kendrick N Kay. Improving the accuracy of single-trial fmri response estimates using gimsingle. Elife, 1 11:777599, 2022.
Ruijic Quan, Wenguan Wang, Zhibo Tian, Fan Ma, and Yi Yang. Psychometry: An omnifit model for image reconstruction from human brain activity. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 233-244--0000 243, 2024.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen, Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 1(2):3, 2022.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition, pp. 10684-10695, 2022.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Bureu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. Advances in neural informa- tion processing systems, 35:36479-36494, 2022.
Paul Scotti, Atmadeep Banerjee, Jimmie Goode, Stepan Shabalin, Alex Nguyen, Aidan Dempster, Nathalie Verlinde, Elad Yundler, David Weisherg, Kenneth Norman, et al. Reconstructing the mind's eye: fmri-to-image with contrastive learning and diffusion priors. Advances in Neural Information Processing Systems, 36, 2024a.
Paul Scotti, Mihir Tripathy, Cesar Kadir Torrico Villanueva, Reese Kneeland, Tong Chen, Ashutosh Narang, Charan Santhirasegaran, Jonathan Xu, Thomas Naselaris, Kenneth A Norman, et al. Mindeye2: Shared-subject models enable fmri-to-image with 1 hour of data. arXiv preprint arXiv:2403. 11000 2024b.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruy Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE international conference on computer vision, pp. 618-626, 2017.
Justine Sergent, Shinsuke Ohta, and Brennan Macdonald. Functional neuroanatomy of face and object processing: positron emission tomography study. Brain, 115(1):15-36, 1992.
Guobin Shen, Dongcheng Zhao, Xiang He, Linghao Feng, Yiting Dong. Jihang Wang. Qian Zhang, and Yi Zeng. Neuro-vision to language: Image reconstruction and language enabled interaction via brain recordings. CoRR, 2024.
13

---

Published as a conference paper at ICLR 2025
Guobin Shen, Dongcheng Zhao, Xiang He, Linghao Feng, Yiting Dong. Jihang Wang. Qian Zhang, and Yi Zeng. Neuro-vision to language: Enhancing brain recording- bbstn visual reconstruction and language interaction. Advances in Neural Information Processing Systems, 37:98083-98110, 2025.
Xiaoyu Shi, Zhaoyang Huang, Fu-Yun Wang, Weikang Bian, Dasong Li, Yi Zhang, Manyuan Zhang, Ka Chun Cheung, Simon See, Hongwei Qin, et al. Motion-i2v: Consistent and controllable image-to-vided generation with explicit motion modeling. In ACM SIGGRAPH 2024 Conference Papers, pp. 1-11, 2024.
Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International conference on machine learn- ing. pp. 2256-2265, PMLR, 2015.
Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 2020.
Anthony Stigliani, Kevin Weiner, and Kalanit Grill-Spector. Temporal processing capacity in high-level visual cortex is domain specific. Journal of Neuroscience, (5(36):12412-1244) 2015.
Christian Szegedy, Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826 2016.
Mingxing Tan and V Le Quoc. Efficientnet: Rethinking model scaling for convolutional neural networks, september 2020. arXiv preprint arXiv:1905.1194
Jerry Tang, Meng Du, Vy Vo, Vasudev Lal, and Alexander Huth. Brain encoding models based on multimodal transformers can transfer across language and vision. Advances Neural Information Processing Systems, 36, 2023.
Aaron Van Den Oord, Oriol Vinyals, al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
Ashish Vaswani, Noam Shazeer, and Niki Parmar. Attention is all you need. Advances in Neural Information Processing Systems, 2017.
Aria Y Wang, Kendrick Kay, Thomas Naselaris, Michael Tarr, and Leila Wehbe. Better models of human high-level visual cortex emerge from natural language supervision with large and diverse dataset. Nature Machine Intelligence, 5(12): 1415):4415 2023.
Shizun Wang, Songhua Liu, Zhenxiong Tan, and Xinchao Wang. Mindbridge: A cross-subject brain decoding framework. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 11333-11 11342, 2024.
Zhou Wang, Alan C Bovik, Hamid R Sheikh, and Eero P Simoncelli. Image quality assessment: from error visibility structural similarity. IEEE transactions on image processing, 13(4):600- 612, 2004.
Weihao Xia, Raoul de Charette, Cengiz Oztireli, and Jing-Hao Xue. Dream: Visual decoding from reversing human visual system. In Proceedings of the IEEE/CVF Winter Conference on Applica- tions of Computer Vision. pp. 8226-8235, 2024.
Xingqian Xu, Zhangyang Wang, Gong Zhang, Kai Wang, and Humphrey Shi. Versatile diffusion: Text, images and variations all in one diffusion model. In Proceedings of the IEEF/CVF Interna- tional Conference on Computer Vision, pp. 7754-7765, 2023.
Huzheng Yang, James Gee, and Jianbo Shi. arXiv:2308.00 01 175, 2023.
Memory encoding model. arXiv preprint
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. In Proceedings the IEEE/CVF International Conference on Computer Vision, pp. 2023.
14

---

Published as a conference paper at ICLR 2025
A
ADDITIONAL IMPLEMENTATION DETAILS
A.1 ADDITIONAL IMPLEMENTATION DETAILS ON MINDSIMULATOR
fMRI Autoencoder. The autoencoder consists of voxel encoder and voxel decoder. The voxel encoder first maps fMRI (12682-15724 voxels, varies across subjects) into a 256-dimensional la- tent space using ridge regression. It then further processes the embedding in the latent space using 2 blocks, each containing multiple Linear layers, LayernNorm layers, activation layers, and resid- ual connections. Finally, linear projector projects the 256-dimensional latent embedding to the high-dimensional fMRI representation of 257 768. And the voxel decoder arranges the three com- ponents exactly in reverse. The fMRI autoencoder is trained end-to-end under the supervision of MSE loss and SoftCLIP loss, with both losses weighted equally at 1.
We also conduct an ablation study on the fMRI autoencoder, investigating the effects of latent space dimensions, the number of residual blocks, and whether end-to-end training is performed (if not end-to-end, we first use SoftCLIP supervision to learn the joint latent space and then train the voxel decoder). The ablation results are shown in Table 5 where we evaluate the MSE between ground truth voxel and encoding-then-decon then-decoding voxel (voxel input) as well as the MSE between ground truth voxel and voxel decoded form image representations (image input). We find that increasing the latent space dimensions and the number of residual blocks does not necessarily lead to performance improvements, so we ultimately opt for a smaller configuration to reduce memory overhead. Addi- tionally, we discover that end-to-end training results lower MSE of voxel input. Furthermore, the extreme MSE metries of image input indicate that IMRI representation and image representation are disjointed, underscoring the necessity of introducing a diffusion estimator.
MSE
lidden dim
blocks
End-to-End
voxel input
image input
256
V 0.227
V
7.470
256
0.240
8.305
512
<
0.252
5.295
256
X 0.401
1.687
256
X 0.411
2.405
512
X 0.343
8.124
Table 5: Ablation Results for fMRI Autoencoder.
Diffusion Estimator. The diffusion estimator contains 6 Transformer blocks. The inputs are 257 image tokens, 257 noised iMRI tokens, and time embedding. Its output is 257 denoised fMRI tokens. We add absolute positional embeddings to the noised fMRI tokens and do not use learn- able query tokens, because this significantly saves on memory. As for attention, we simply use bidirectional attention instead of causal attention.
A.2 ADDITIONAL DETAILS ON EVALUATION METRICS
Decoding Model. We the trained MindEye2 (Scotti et 2024b) our decoding model for rec- ognizing the visual semantics contained in synthetic fMRI. At this stage, all decoding models have adopted a similar approach. They first utilize contrastive learning align IMRI voxels with the im- age latent space, then transform the IMRI representation into the image representation. Finally, the transformed IMRI representation
used as the text condition input for trained T2I models (Rom-l bach et al. 2022 Xu et al., 2023) to reconstruct the image. Obviously, due to encoding as the reverse process of decoding, our MindSimulator has been greatly inspired by those decoding models.
Image Reconstruction Metrics. PixCorr denotes the pixel-wise correlation between ground truth image and reconstruction results. SSIM denotes Structural similarity index metric (Wang et al. 2004) between ground truth and reconstructions. Alex(2), Alex(5), Incep, and CLIP are metrics that refer to two-way identification (chance = 50%) using different models. Alex(2) denotes two- way comparisons are performed with the second layer of AlexNet (Krizhevsky et al. (2012), Alex(S) denotes comparisons with the fifth layer of AlexNet, Incep denotes comparisons with the last pooling layer of Inception V3 (Szegedy of CLIP ViT-L/14 (Radford et al. 2021). Two-way identification refers to percent correct across comparisons gauging if the original image embedding more similar to its paired voxel embedding
al.) 2016), and CLIP denotes comparisons with the final layer
15

---

Published as a conference paper at ICLR 2025
randomly selected voxel embedding. Eff and SwAV refer to the average correlation distance with EfficientNet-Bi (Tan & Quoc) and SwAV-ResNet50 (Caron et al. 2020). The average time cost for evaluating single 1MRI is within 1-2 seconds.
A.3 DETAILS ON BASELINES
Linear Regression Encoding. The linear regression encoding model consists of two linear layers, the first of which projects the 257 x 768 image representation directly into the 2048 latent space, and the second of which projects the latent into the voxel dimension (such as 15724). We train this linear model for 150 epochs using Adam W, and the learning rate decreases linearly from 3e-4. For inference, we use the checkpoint with the smallest MSE metric in the validation set.
Transformer Encoding, The Transformer Encoding model uses 10 ROI queries. Each ROI query a learnable token, corresponding ROL. We consider total of 10 ROIs, including lh-faces, Ih- bodies, Ih-places, wh-words, Ih-remains, rh-faces, rh- th-bodies, rh-places, rh- rh-words, and rh-remains. single-layer Transformer with one cross-
The 10 ROI queries and image tokens input attention and one self-attention operation. Each output ROI query is then mapped using a single linear layer to fMRI voxel. The predicted voxel is then multiplied by mask that is zero everywhere except for the vertices belonging to that ROI, ensuring that the gradient feeding back from the loss will only train linear mappings the voxel of the queried ROI. During inference, the predicted voxel from different ROI readouts will then be combined using the same masks to generate the synthetic fMRL. We train this encoding model for 150 epochs using AdamW, and the learning rate decreases linearly from 3e-4. Finally, the checkpoint with the smallest MSE metric is used.
A.4 ADDITIONAL DETAILS ON LOCALIZATION
Images. We used the MSCOCO images adopted in the NSD experiment (a total of 73,000 images, including 9,000 images unique each subject and 1,000 images shared among subjects). Note that each subject had fMRI data corresponding to only 10,000 images.
Prompts. In the NSD) fLoc experiments, researchers select visual stimuli from fixed categories. Specifically, places-stimuli contain "house" and "hallway", bodies-stimuli contain human "body" and "limb", faces-stimuli contain real "adult face" and "children face", and words-stimuli con- tain "characters" and "numbers". Therefore, to validate our localization with places- bodies, faces- and words-selective regions, we utilize the following prompts for zero-shot classifica- tion: ["houses or corridors", "human bodies or human limbs", "real human faces", "words or numbers"]. As for the exploration of novel concept-selective regions, we use the most common prompts, ["a photo of surfer", "a photo of plane", "a photo of food", "a photo of bed"].
Significance Thresholds. In the NSD fLoc experiments, the significance threshold is set to O, which very generous. this setting, the range concept-selective regions is very large, for example, Subj06's s bodies-selective region even contains 4887 voxels. In our concept-selective voxel localization, the significance factor of the t-eest is set to 0.01. In this setting, the selected region usually contains voxels ranging from hundreds to few thousand.
16

---

Published as a conference paper at ICLR 2025
B ADDITIONAL RESULTS
B.1 ADDITIONAL METRICS FOR SYNTHETIC FMRI
We first include R-squared f², which is commonly used in neuroscience studies. The results are presented in Figure TO We also evaluated the synthetic fMRIs with additional metrics covering the similarity of functional connectivity. spatial structure similarity, and the similarity of fMRI Repre- sentations. The evaluation metrics are described follows.
Similarity of Functional Connectivity Graphs. We use the functional areas provided by NSD. Dimensionality reduction is performed on the voxels within each ROI using PCA to ensure they have the same length. Subsequently, Pearson correlation was calculated among the different regions to obtain the functional connectivity maps. The similarity between the functional connectivity maps of the synthetic and real IMRI data is then evaluated using the mean absolute error.
Spatial Gradients. We compute the spatial gradients of each sample in the test set across all di- rections and then calculate the absolute difference between the ground-truth fMRI and the synthetic fMRL. The average results of each voxel are reported.
Similarity of IMRI Representations. we compute the two-way classification accuracy of fMRI representations.
All results are reported Table 6
It can be observed that the results of different methods are similar.
Methods
Connectivity Graphs
Sputial Gradients..
fMRI Rep. Similarity?
Lincar Encoding Model
0.127466
0.536810
99.27%
Transformer Encoding Model MindSimulator
0.123541
0.543872
99 99.43%
0.122895
0.538561
99.77%
Table 6: Additional metrics for synthetic fMRI.
Subj01
Subj02
Subj05
Subj07
Figure 10: The R metric of synthetic fMRI for each subject.
B.2 ADDITIONAL VISUALIZATION OF SYNTHETIC FMRI
In addition, we further provide the reconstructed image from the synthetic fMRI for 4 subjects, as shown in Figure I to Figure
B.3 3 ADDITIONAL RESULTS ON EXPLORING NOVEL REGIONS
We display the cortical selective regions the same concepts among different subjects. As illus- trated in Figure 16 to Figure 18, despite subtle differences, the activation of the same concepts in different subjects revealed similar patterns, with the strongest areas of activation typically occurring in the same locations. This indicates high degree of similarity among individual brains.
17

---

Published as a conference paper at ICLR 2025
GT
Synthetic GT Synthetic
STOP
STOOP
Figure 11: Additional visualization of reconstructed images from Subj01 synthetic fMRL. GT seen
visual stimuli. Synthetic reconstructed images from synthetic fMRI. Randomly selected.
GT
Synthetic
Synthetic GT
Figure 12: Additional visualization of reconstructed images from Subj02 synthetic fMRL. = seen
visual stimuli. Synthetic reconstructed images from synthetic IMRL. Randomly selected.
18

---

Published as a conference paper at ICLR 2025
GT
Synthetic
Synthetic GT
Figure 13: Additional visualization of reconstructed images from Subj05 synthetic fMRL. =een seen
visual stimuli. Synthetic reconstructed images from synthetic fMRL. Randomly selected.
GT
Synthetic
Synthetic GT
Figure 14: Additional visualization of reconstructed images from Subj07 synthetic fMRL. GT= seen
visual stimuli. Synthetic reconstructed images from synthetic fMRL. Randomly selected.
19

---

Published as a conference paper at ICLR 2025
Subj01
Subj02
Subj05
Subj07
Figure 15: Surfer-selective regions of each subject.
Subj01
Subj02
Subj05
Subj07
Figure 16: regions of each subject.
20

---

Published as a conference paper at ICLR 2025
Subj01
Subj02
Subj05
Subj07
Figure 17: Food-selective regions of each subject,
Subj01
Subj02
Subj05
Subj07
Figure 18: Bed-selective regions of each subject.
21

---

Published as a conference paper at ICLR 2025
C VALIDITY OF SEMANTIC METRICS
The question whether semantic metrics can effectively identify the semantics contained in fMRI signals is one that requires further verification. We explore this through noise decoding experiments. We use three types of noise as the input of the trained decoding model (MindEye22. The results are shown in Table 7
Method
PixCorrt
SSIMT
Alex(2)t
Alex(S)1
Incept
CLIPt
Em]
SwAV
Random Input
0.027
0.178
50.7%
50.0%
50.8%
50.1%
0.980
0.632
Shuffled fMRI
0.023
0.143
50.4%
51.0%
50.4%
50.2%
0.983
0.642
Shuffled Synthetic IMRI
0.021
0.114
50.4%
51.4%
49.9%
50.7%
0.986
0.650
MindSimulator
0.201
0.298
89.6%
96.8%
93.2%
91.2%
0.688
0.393
Table
Validity experiments semantic metrics.
It can be seen the decoding performance of shuffted IMRI is comparable to that of random input. Most performance of baselines reach their theoretical minimum values, such Pearson correla- tion and pixel similarity (PixCorr) approaching O: Alex(2), Alex(5), Incep, and CLIP nearing 50%. These results suggest that trained decoding models can successfully capture the visual semantics contained in fMRI rather than other irrelevant content. At the same time, this also indicates that re- construction can only be achieved when the fMRI contains real visual semanties. We further display the reconstructed results of noise input. As shown in Figure 19 no clear visual semantics can be reconstructed from noise.
Reconstructed results from noise. No clear visual semantics are contained in the recon- structed images. Please zoom in for better viewing.
Figure 19:
D COMPARISON OF DIFFERENT CONCEPT-SELECTIVE REGION LOCALIZATION METHODS
In concurrent papers, Shen al. (2055) proposed a concept-selective region localization method
based on Grad- CAM (Selvaraju et al.) 2017). We briefly compare it with our method, and the corresponding results are shown in Figure 20
Shen
Ours
Shen et
Ours
Cow
Kitehen
Tree
Road
J
Figure 20: Comparison between Grad-CAM-basee localization and method. Our method local- izes the concept-selective regions more focused on the higher-level visual cortex.
22

---

Published as a conference paper at ICLR 2025
ETHIC STATEMENT
Our research adheres to the ICLR Code of Ethics. All experiments in this paper are conducted using open-source datasets, and no potential ethical concerns are associated with this work.
REPRODUCIBILITY STATEMENT
This study employs generative encoding models synthesize fMRI data, facilitating the localization of concept-selective regions. All preprocessed data, code, and model parameters used in our research will be made publicly available upon publication. Detailed protocols for data preprocessing, model training. and evaluation have been provided in our manuscript, enabling independent reproduction.
FUTURE WORK
In future work, we will validate the effectiveness of MindSimulator more fMRI datasets and verify the validity of data-driven concept selection region localization through neuroscience ex- periments. We will also conduct more detailed exploration of the concept selection regions using synthetic fMRL, develop a human brain concept map, and study individual differences. Addition- ally, we exploring the potential of encoding model feedback to improve cross-subject decoding performance using synthetic fMRI.
23


Reviews:
[{"review_id": "blhbbyeZ6V", "reviewer": "Reviewer_YLHx", "summary": "The paper presents a new data-driven approach to localize concept-selective regions in the brain by using synthetic brain recordings generated via a probabilistic model, MindSimulator, conditioned on concept-oriented visual stimuli. This approach enhances coverage and reduces bias, achieving high prediction accuracy in localizing known concept regions.", "questions": "1 - Given the brain’s dynamic complexity and somewhat chaotic behavior, generative models offer both benefits and limitations in modeling brain function. Could the authors evaluate the similarity in temporal and spatial gradients between the original and synthetic data to better assess these dynamics?\n\n2 - Additionally, it would be valuable if the authors could quantify the similarity in functional connectivity maps between the original and synthetic data at each timepoint as well.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1 - The authors employ a generative fMRI encoding model to synthesize individual fMRI signals corresponding to concept-oriented visual stimuli, addressing the inherent one-to-many correspondence issue between visual stimuli and fMRI recordings.\n\n2 - The paper is well-structured, with a clear formulation of the problem and a thorough description of the proposed model's components and methodology.\n\n3 - The authors provide extensive ablation studies that effectively validate the model architecture's contributions and showcase the performance impact of each component.", "weaknesses": "1 - Capturing both spatial and temporal dependencies within the autoencoder is essential for producing meaningful representations of brain activity, which is inherently dynamic. The current model appears to underutilize temporal information based on the description in Supplementary Material Section A. To address this limitation, you might consider adding recurrent layers, such as LSTMs or GRUs, or using 3D convolutions in the autoencoder to enhance temporal processing. Additionally, it would be helpful to clarify in the main text or supplementary materials if and how temporal dependencies are integrated in the current approach. This added information would improve understanding of how well the model aligns with the time-varying nature of fMRI data.\n\n2 - It would be helpful if the authors could clarify their voxel selection and masking process, specifically how spatial relationships between neighboring voxels are preserved when creating the autoencoder input. If there is a risk of losing local spatial context, consider alternative approaches, such as using 3D convolutions or patch-based inputs, which may mitigate this issue and maintain spatial continuity within the masked regions.\n\n3 - To improve the evaluation of your results, please include comparisons with specific, relevant works. For example, you may consider applying a connectivity-based parcellation approach (ref are given below) to both the original and synthetic data to examine whether similar visual networks emerge in each case. Including these comparisons would help readers to contextualize the reported metrics and enable a clearer understanding of your model's relative performance and its contributions to the field.\n\n[1] Du, Y., Fu, Z., Sui, J., Gao, S., Xing, Y., Lin, D., ... & Alzheimer's Disease Neuroimaging Initiative. (2020). NeuroMark: An automated and adaptive ICA based pipeline to identify reproducible fMRI markers of brain disorders. NeuroImage: Clinical, 28, 102375.\n\n[2] Vu, T., Laport, F., Yang, H., Calhoun, V. D., & Adalı, T. (2024). Constrained independent vector analysis with reference for multi-subject fMRI analysis. IEEE Transactions on Biomedical Engineering.\n\n4 - To aid in assessing scalability, please provide details on the computational complexity of the model, including training time, memory usage, and the hardware specifications used in your experiments. These details would offer valuable insight into the practical feasibility of implementing your approach in various research or clinical settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EUED5KMgIc", "reviewer": "Reviewer_KVAs", "summary": "This is a paper with interesting results. However, I am uncertain whether the technique presented represents a major advancement. A key gap in the literature appears to be the investigation of cognitive processes, including the concepts discussed by the authors, not only in a spatial context but also in terms of activation trajectories over time. The authors have already addressed this point in their discussion with another reviewer. While I agree that NSD does not necessarily contribute additional insights in this context, there are datasets on movie viewing with corresponding ratings that could have been beneficial for this study.", "questions": "Nothing anymore. the authors have done a great job.", "limitations": "", "rating": 5, "confidence": 2, "soundness": 1, "presentation": 1, "contribution": 1, "strengths": "-", "weaknesses": "A key gap in the literature appears to be the investigation of cognitive processes, including the concepts discussed by the authors, not only in a spatial context but also in terms of activation trajectories over time. I am not sure if the current paper provides a systematic solution.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fjkO3u0LP2", "reviewer": "Reviewer_SRDC", "summary": "This paper proposes a new way to implement concept-localization in the brain using a learned generative model which synthesizes fMRI responses. This is derived from the observation that fMRI responses to the same stimuli can be noisy and are better captured by sampling from a random variable instead of a learned (discriminative/static) model. A latent representation is jointly learned via CLIP in which an image embedding is paired with a voxel embedding and then trained according to the SoftCLIP loss.\n\nThe authors show reconstruction is possible via their proposed method to use synthetic fMRI, but the authors fail to show that the brain data could just as easily be ignored. Image encodings are passed into the sampler and decoders are highly able to create very realistic images but it's not clear that the modelling of the resting state inputs and learned fMRI is actually doing anything useful in a clear way as the authors make it seem (with huge swaths of cortex claimed for very restrictive conceptual categories). There are arguments for how discrete these are but once you expand the classes beyond the very limited amount presented, then quantified overlap, it would be clear that many patches are not conceptually distinct. That's my assumption. \n\nThe idea is an interesting one, but the lack of good experimental testing against strong baselines (particularly, testing with shuffled/random fMRI data). The bulk of the promise shown here might actually be just by going between image embeddings (via the voxel encoder as it was jointly trained on image representations and not fMRI data alone).", "questions": "The introduction raises some claims that need supporting evidence. How do you know that the efforts to go into designing common functional localization images are insufficient and poorly generalizable? What promotes this observation? Why should a functional localiser be embedded within a naturalistic scene? Naturalistic stimuli are famously confounded across multiple dimensions and the artificial placement of a core concept in a bare background is a method to remove potential confounds. Yes, it’s undesirable because our vision is based around naturalistic scenes, but the argument for naturalistic images in functional localization is only inviting trouble. We would lose specificity and be more unlikely to be sure that we’re not detecting confounding background information and mistakenly attributing brain activity to core concepts within the images. The arguments as they’re outlined don’t naturally follow on from one another in this exposition of the paper’s contributions.\n\n- Line 157: do you mean for the comma to be a subtraction symbol in the MSE equation?\n- Why do you start the inference sampler with resting state fMRI data? What's the idea here? It's not really explained in Section 3.4.\n- If an amended version is submitted, could you put the ROI boundaries on your flatmaps to better orient the distributions of voxel encodings?\n- In 6.1 what's going on here? Are you using MSCOCO images or images for which there is fMRI data in NSD? It's not clear\n- Also in 6.1, you mention the t-test that is done voxelwise, where are these results? What was the threshold? I don't really understand what you've done or how you have set up your test and there is also no mention of multiple comparisons correction (big red flag for me).", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper has a pretty good grasp on the recent literature and various approaches that have been experimented on in this area, showing a wide depth of knowledge. The analyses seem detailed and it's clear a lot of work went into some parts of the experimental analysis. There will be a pretty detailed Weaknesses section, but it's easier to point out identified weaknesses than identify lists of things done correctly. I do have a fair few issues with the way the analysis was done, but I think with some tweaks and additional analyses that are robust against better controls, better description, this paper does have potential.", "weaknesses": "* Captions to figures are mostly vacuous and non-descriptive and need to be expanded to better describe the associated figures\n* Some points are argued but are presented without evidence and for the kind of statements they are, definitely require a solid backing (see Questions section)\n- Citation format is not consistent. Many citations should be in parentheticals but are not (needs to be fixed for camera-ready version)\n- The language is overly flowery in a way that makes the claims nonsensical (e.g. “Fortunately, we effectively explore novel concept-selective regions, capably providing explicit hypothesis constraints…”)\n- Language needs to be checked by a person intimately familiar with the conventions of academic written English to correct some unusual and unclear phrasing (in the methods section especially)\n\nThere have been numerous recent works that have highlighted how these types of models can effectively perform the same function when replacing brain data with random noise or brain responses that aren't paired correctly with the same responses. \n\nHuge caveat here that “**if it can be reconstructed, then the fMRI contains the information**” but you can often do reconstruction equally well from random noise, there doesn’t have to be anything real in the fMRI data. Kamitani recently showed this (https://arxiv.org/abs/2405.10078) and this paper also did (https://arxiv.org/abs/2405.06459) with EEG. \n\nThe paper fails to take into account a number of confounds and does not seem to understand just how drastic this aspect of the analysis could be on changing the presented results. You can't present images of food and not take into account that you might be modelling shape (round plates, round food shapes) or lower-level features like colour (food is often colourful). These have been huge issues in the concept localization space using datasets like NSD but I didn't see any citations or awareness of this issue. Also, it's not likely that the concept of \"surfer\" or \"bed\" takes up anywhere near as much cortical territory as some of these plots indicate. There is high-level confounding going on here that undermines the idea of concept localization. This is why the handcrafted stimuli were carefully created in the first place, to avoid this issue. The idea of this paper seems like it goes back in the wrong direction. \n\nThe results in Table 3 during the ablation analysis show often minimal drops when ablating important components of the paradigm, which lead me to believe that confounds and lack of good baselines are hiding shortcut learning and cheats that the model is making use of instead of it being primarily a method centred on good fMRI representations.\n\nIf you have focused on the localizers used in NSD then I think it's important you cite the (ubiquitous) paper that NSD (and many other fMRI datasets) use, namely the fact that these fLoc images come from Stigliani et al. 2015 (https://www.jneurosci.org/content/35/36/12412). \nIt seems quite the oversight to not have cited this given the content of the submission, especially as you're using the images from this paper in the figures of your dataset.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "a94hnlV5fA", "reviewer": "Reviewer_Atef", "summary": "This paper proposes \"MindSimulator\", a framework that synthesizes fMRI data based on visual stimuli through an fMRI autoencoder, diffusion estimator, and inference sampler. The authors first assessed the performance of the fMRI autoencoder and diffusion estimator using various metrics, demonstrating their capability to generate high-quality fMRI data. They then used the synthesized fMRI data to explore correlations between manually selected images and brain activity, offering new insights for neuroscience research.", "questions": "1. In the *Inference Sampler* section, the mention of \"resting-state brain activity fMRI\" could be misleading, suggesting that the model can generate resting-state fMRI data. However, I could not find evidence of any relevant dataset being used. Could the authors clarify this point?\n2. In the *Out-of-Distribution Generalization* section, CIFAR-10/100 was used, and metrics were calculated based on images decoded from synthesized fMRI data. As noted in the weaknesses, this approach may introduce bias. Why not use an image-fMRI dataset, like THING-fMRI, to compute metrics directly on fMRI data?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "1. The paper offers a novel perspective by applying well-established fMRI visual decoding models for fMRI signal synthesis, with thorough validation to demonstrate reliability.\n2. This study introduces a new tool for exploring *concept-selective regions*, significantly enhancing the flexibility of investigating how specific human visual representations of concepts are spatially distributed in the brain.", "weaknesses": "1. The algorithm for localizing concept-selective regions may lack sufficient validation, as the paper only compares this approach to fLoc, without further support from neuroscience literature. Consideration of alternative methods, like Neurosynth or Text2Brain, could strengthen the results, as these methods allow a broader selection of concepts correlated with brain activity, potentially detecting concepts not covered by fLoc. \n\n2. In the *Evaluation Metrics* section, the method of validating generated fMRI data based on the quality of generated images may not be reliable due to its reliance on a separate trained decoding model. Given the complexity of visual decoding from fMRI, this dependence could reduce the robustness of the evaluation. Exploring alternative evaluation methods, such as comparing generated data with latent representations in the voxel encoder’s latent space, might provide more direct validation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)