Brain Diffusion for Visual Exploration: Cortical Discovery using Large Scale Generative Models
Authors: Andrew Luo
Margaret Marie Henderson
Leila Wehbe
Michael J. Tarr
Venue: NeurIPS 2023 oral (2023)
Keywords: neuroscience
brain
fmri
generative models
diffusion models
image synthesis
visual cortex

Abstract: A long standing goal in neuroscience has been to elucidate the functional organization of the brain. Within higher visual cortex, functional accounts have remained relatively coarse, focusing on regions of interest (ROIs) and taking the form of selectivity for broad categories such as faces, places, bodies, food, or words. Because the identification of such ROIs has typically relied on manually assembled stimulus sets consisting of isolated objects in non-ecological contexts, exploring functional organization without robust a priori hypotheses has been challenging. To overcome these limitations, we introduce a data-driven approach in which we synthesize images predicted to activate a given brain region using paired natural images and fMRI recordings, bypassing the need for category-specific stimuli. Our approach -- Brain Diffusion for Visual Exploration ("BrainDiVE") -- builds on recent generative methods by combining large-scale diffusion models with brain-guided image synthesis. Validating our method, we demonstrate the ability to synthesize preferred images with appropriate semantic specificity for well-characterized category-selective ROIs. We then show that BrainDiVE can characterize differences between ROIs selective for the same high-level category. Finally we identify novel functional subdivisions within these ROIs, validated with behavioral data. These  results advance our understanding of the fine-grained functional organization of human visual cortex, and provide well-specified constraints for further examination of cortical organization using hypothesis-driven methods.

Brain Diffusion for Visual Exploration: Cortical Discovery using Large Scale Generative Models
Andrew Luo Carnegie Mellon University afluo @ cmu.edu
Margaret M. Henderson Carnegie Mellon University mmhender@ cmu.edu
Leila Wehbe Carnegie Mellon University Iwchbe cmu.edu
Michael J. TTarr" Carnegie Mellon University michaeltarr cmu.edu
Abstract
A long standing goal in neuroscience has been to elucidate the functional or- ganization of the brain. Within higher visual cortex, functional accounts have remained relatively coarse, focusing on regions of interest (ROIs) and taking the form of selectivity for broad categories such as faces, places, bodics, food, or words. Because the identification of such ROIs has typically relied on manually assembled stimulus sets consisting of isolated objects in non- non-ecological contexts, exploring functional organization without robust a priori hypotheses has been challenging. To overcome these limitations, we introduce a data-driven approach in which synthesize images predicted to activate given brain region using paired natural images and fMRI recordings, bypassing the need for sategory-speccife stimuli. Our approach Brain Diffusion for Visual Exploration ("BrainDiVE"), builds on recent generative methods by combining large-scale diffusion models with brain-guided image synthesis. Validating our method, we demonstrate the ability to synthesize preferred images with appropriate semantic specificity for cell-characterized category-selectivy ROIs. We then show that BrainDiVE can characterize differences between ROIs selective for the same high-level category. Finally we identify novel functional subdivisions within these ROIs, validated with behavioral data. These results advance our understanding of the fine- fine-grained func- tional organization of human visual cortex, and provide well-specified constraints for further examination of cortical organization using hypothesis-drive methods. Code and project site: https://www.cs.co cmu, att.e///af/foo/raiiiiiii
Introduction
The human visual cortex plays a fundamental role in our ability to process, interpret, and act on visual information. While previous studies have provided important evidence that regions in the higher visual cortex preferentially process complex semantic categories such as faces, places, bodies, words, and food [1, 2, 3, 4, 5, 6, 71. these important discoveries have been primarily achieved through the use of researcher- crafftte stimuli. However, hand-selected, synthetic stimuli may bias the results or may not accurately capture the complexity and variability of natural scenes, sometimes leading to debates about the interpretation and validity of identified functional regions [8]. Furthermore, mapping selectivity based responses fixed set of stimuli is necessarily limited, in that it can only identify selectivity for the stimulus properties that are sampled. For these reasons, data-driven methods for interpreting high-dimensional neural tuning are complementary traditional approaches.
We introduce Brain Diffusson for Visual Exploration ("BrainDiVE"), a generative approach for synthesizing images that are predicted to activate given region in the human visual cortex. Several
Co-corresponding Authors
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

FWOH CLO
SUPRM
EL EL. 10
Face regions
Place regions
Body regions
Word regions
Food regions
Figure 1: Images generated using BrainDiVE. Images are generated using a diffusion model with maximization of voxels identified from functional localizer experiments as conditioning, We find that brain signals recorded via fMRI can guide the synthesis of images with high semantic specificity, strengthening the evidence for previously identified category selective regions. Select images are shown, please see below for uncurated images.
recent studies have yielded intriguing results by combining deep generative models with brain guidance [9, 10, 11]. BrainDiVE, enabled by the recent availability of large-scale fMRI datasets based on natural scene images [12, 13], allows us to further leverage state-of-the-art diffusion models in identifying fine-grained functional specialization in an objective and data-driven manner. DiV is based on image diffusion models which are typically driven by text prompts in order to generate synthetic stimuli [14]. We replace these prompts with maximization of voxels in given brain areas. The result being that the resultant synthesized images are tailored targeted regions in higher-order visual areas. Analysis of these images enables data-driven exploration of the underlying feature preferences for different visual cortical sub-regions. Importantly, because the synthesized images are optimized to maximize the response of a given sub-region, these images emphasize and isolate critical feature preferences beyond what was present in the original stimulus images used in collecting the brain data. To validate our findings, we further performed several human behavioral studies that confirmed the semantic identities of our synthesized images.
More broadly, we establish that BrainDiVE can synthesize novel images (Figure 1) for category- selective brain regions with high semantic specificity. Importantly, further show that Brain- DiVE can identify ROI-wise differences in selectivity that map to ecologically relevant properties. Building this result,
able to identify novel functional distinctions within sub-regions of existing ROIs Such results demonstrate that BrainDiVE can be used in data-driven manner to enable new insights into the fine-grained functional organization of the human visual cortex.
2 Related work
Mapping High-Level Selectivity in the Visual Cortex. Certain regions within the higher visual cortex are believed to specialize in distinct aspects of visual processing, such as the perception of faces, places, bodies, food, and words [15, 3, 4, 1, 16, 17, 18, 19, 5, 20]. Many of these discoveries rely on carefully handcrafted stimuli specifically designed to activate targeted regions. However, activity under natural viewing conditions known to different [21]. Recent efforts using artificial neural networks image-computale encoders/predictooss of the visual pathway [22, 23, 24, 25, 26, 27, 28, 29, 30] have facilitated the use of more naturalistic stimulus sets. Our proposed method incorporates image-computable encoding model in line with this past work.
Deep Generative Models. The recent rise of learned generative models has enabled sampling from complex high dimensional distributions. Notable approaches include variational autoencoders [31, 32). generative adversarial networks [33], flows [34, 35], and score/energy/diffssionn models [36, 37, 38, 39.. It is possible to condition the model on category [40, 41], text [42,43], or images [44]. Recent
diffusion models have been conditioned with brain activations reconstruct observed images [45,
46, 47, 48, 49]. Unlike BiainDiVE, these approaches tackle reconstruction but not synthesis of novel images that are predicted to activate regions of the brain.
Crain-Conditioned Image Generation. The differentiable nature of deep encoding models in- spired work create images from brain gradients in mice, macaques, and humans [50, 51, 52]. Without constraints, the images recovered are not naturalistic. Other approaches have combined deep generative models with optimization to recover natural images in macaque and humans [10, 11, 9]. Both [11, 9] utilize fMRI brain gradients combined with ImageNet trained BigGAN. In particu- lar [11] performs end-to-end differentiable optimization by assuming a soft relaxation over the 1, 000 ImageNet classes; while [9] trains encoder the NSD dataset [13] and first searches for

---

Diffusion models enable sampling from a data distribution p(z) by iterative denoising. The top-classes, then performs gradient optimization within the identified classes. Both approaches are restricted to ImageNet images, which are primarily images of single objects. Our work presents major improvements by enabling the use of diffusion models [44] trained on internet-scale datasets [53] over three magnitudes larger than ImageNet. Concurrent work by [54] explore the use of gradients from macaque V4 with diffusion models, however their approach focuses on early visual cortex with grayscale image outputs, while our work focuses on higher-order visual areas and synthesize complex compositional scenes. By avoiding the search-based optimization procedures used in [9], our work not restricted to images within fixed class in ImageNet. Further, to the authors' knowledge we the first work to image synthesis methods in the identification of functional specialization in sub-parts of ROIs.
X2
1
Latent Decoder
image
Brain Encoder
RSC
OPA
Denoiser
PPA
gradient flow
Null text
Brain region target
||Gradient] 1
RGB Image
Start
End
Figure 2: Architecture of brain guided diffusion (BrainDiVE). Top: Our framework consists of two core components: (1) A diffusion model trained to synthesize natural images by iterative denoising; we utilize pretrained LDMs. (2) An encoder trained to map from images to cortical activity. Our framework can synthesize images that are predicted activate any subset of voxels. Shown here are scene-selective regions (RSC/PPA/OPA) on the right hemisphere. Bottom: We visualize every steps the magnitude of the gradient of the brain w.r.t. the latent and the corresponding "predicted ro" [55] when targeting scene selective voxels in both hemispheres. We find clear structure emerges.
3 Methods
We aim to generate stimuli that maximally activate given region in visual cortex using paired natural image stimuli and fMRI recordings. We first review relevant background information on diffusion models. We then describe how we can parameterize encoding models that map from images to brain data. Finally, we describe how our framework (Figure 2) can leverage brain signals as guidance to diffusion models to synthesize images that activate target brain region.
3.1 Background on Diffusion Models
sampling process starts with 2T (N,0, II), and produces progressively denoised samples JT 22- ... until a sample xo from the target distribution is reached. The noise level varies by timestep 6, where the sample at each timestep is weighted combination of zo and e N I)), with 2; = Varzo + ev1 at. The value of a interpolates between N I) and p(z).
In the noise prediction setting, an autoencoder network (0 t) is trained using mean- mrean-squared errr E(a.e.) [1lco(02,)) t) - <|]3] In practice, we utilize a pretrained latent diffusion model (L.M) [44], with learned image encoder Es and decoder Dn. which together act an autoencoder I s3 Do(Eg Eg (Z)). The diffusion model is trained to sample To from the latent space of Ez.
3.2 Brain-Ennn Encoding Model Construction
learned voxel-wise brain encoding model is a function Mg that maps an image I E R3xHXW to the corresponding brain activation fMRI beta values represented as N element vector B E R (T) = B. Past work has identified later layers neural networks as the best predictors of higher visual cortex [30, 56]. with CLIP trained networks among the highest performing brain

---

encoders [28, 57]. As our target is the higher visual cortex, we utilize two component design for our encoder. The first component consists of a CLIP trained image encoder which outputs K dimensional vector as the latent embedding. The second component is linear adaptation layer W E RNNK, E RNV, which maps euclidean normalized image embeddings to brain activation.
CLIP CLIP (2)l2
B (Z W
Optimal W*, by are found by optimizing the mean squared error loss over images. We observe that use of normalized CLIP embedding improves stability of gradient magnitudes w.r.t. the image.
3.3 Brain-Guided Diffusion Model
BrainDiVE seeks to generate images conditioned on maximizing brain activation in a given region. In conventional lext-conditioned diffusion models, the conditioning done one of ways. The first approach modifies the function éo to further accept conditioning vector c, resulting in cø( 4 c). The second approach uses a contrastive trained image-to-conee encoder, and seeks to maximize similarity measure with text-to-concept encoder.
Conditioning on activation of brain region using the first approach presents difficulties. We do not know a priori the distribution of other non-targeted regions in the brain when target region is maximized. Overcoming this problem requires us either have prior p(B) that captures the joint distribution for all voxels in the brain, to ignore the joint distribution that can result in catastrophic effects, or to use a handcrafted prior that may be incorrect [47]. Instead, we propose condition the diffusion model via our image-to to- brain encoder. During inference we perturb the denoising process using the gradient of the brain encoder maximization objective, where 7 a scale, and S C N are the set of voxels used for guidance. We seek to maximize the average activation of S predicted by Mg:
Ethata v 0z V Dz
Σ Mo(Da(z))))
theta
S
Like [14, 58, 59]. we observe that convergence using the current denoised 2; is poor without changes to the guidance. This is because the current image (latent) is high noise and may lie outside of the natural image distribution. We instead use weighted reformulation with euler approximation [55, 59] of the final image:
Va (x. vI (1E.)
x (vi 0n (1 oz)ar
By combining an image diffusion model with a differentiable encoding model the brain, we are able to generate images that seek to maximize activation for any given brain region.
Results
In this section, use BrainDiVE to highlight the semantic selectivity of pre-identified category- selective voxels. We then show that our model can capture subtle differences in response properties between ROIs belonging to the same broad sategory-selective network. Finally, we utilize Brain- DiVE to target finer- finer-grained grained sub-regions within existing ROIs, and show consistent divisions based semantic and visual properties. We quantify these differences in selectivity across regions using human perceptual studies, which confirm that BrainDiVE images can highlight differences in tuning properties. These results demonstrate how BrainDiVE can elucidate the functional properties of human cortical populations, making a promising tool for exploratory neuroscience.
4.1 Setup
We utilize the Natural Scenes Dataset (NSD; [13]), which consists of whole-brain 7T IMRI data from 8 human subjects, 4 of whom viewed 10, 000 natural scene images repeated 3x. These subjects, S1, S2, S5, and S7, are used for analyses in the main paper (see Supplemental for results for additional subjects). All images are from the MS COCO dataset. We use beta- beta- weights (activations) computed using GLMSingle [60] and further normalize each voxel to μ = 0, a = 1 on a per-session basis. We average the fMRI activation across repeats of the same image within subject. The ~9, 000 unique images for each subject ([13)) are used train the brain encoder for each subject, with the remaining 1, 000 shared images used to evaluate R²2 Image generation is per-subject basis and done

---

on an Nvidia V100 using 1, 500 compute hours. As the original category ROIs in NSD are very generous, we utilize stricter category and ROI masks used in our experiments are derived from the logical AND of the official NSD masks with the masks derived from the official /-statistics.
threshold reduce overlap unless otherwise noted. The final
We utilize
which produces images of 512 x 512 resolution using e-prediction. Following best practices, we use multi-step 2nd order DPM-Solver++ [61] with 50 steps and apply 0.75 SAG [62]. We set step size hyperparameter
130.0. Images are resized to 224 x 224 for the brain encoder. (null prompt) used as the input prompt, thus the diffusion performs unconditional generation without brain guidance. For the brain encoder we use ViT-B/10 16, for CLIP probes we use CoCa ViT-L/14. These are the highest performing LAION models of given size provided by OpenCLIP [63, 64, 65, 66]. We train our brain encoders on each human subject separately to predict the activation of all higher visual cortex voxels. See Supplemental for visualization of test time brain encoder R². To compare images from different ROIs and sub-regions (OFA/FFA in 4.3, two clusters in 4.4), we asked human evaluators select which of two image groups scored higher on various attributes. We used 100 images from each group randomly split into 10 non- overlapping subgroups. Each human evaluator performed 80 comparisons, across 10 splits, NSD subjects, and for both IMRI and generated images. See Supplemental for standard error of responses. Human evaluators provided written informed consent and were compensated at $12.00/hour. The study protocol was approved by the institutional review board at the authors' institution.
4.2 Broad Pategory-Selectiiv Networks
bodies
In this experiment, target large groups of category voxels which can encom- pass more than one ROI (Figure 3). These regions have been previously identified as selective for broad semantic categories, and this experiment validates our method using these identified regions. The face-, place-, body-, and word- selective ROIs are identi- fied with standard localizer stimuli [67]. The food-selective voxels were obtained from [5].
faces
places
words
food
ventral
expanded
multiple
Figure
Visualizing category-selective voxels in S1. See text for details on how category selectivity was defined.
The same voxels were used to select the top activating NSD images (referred to as "NSD"" and to guide the generation of BrainDiVE images.
In Figures 4 we visualize, for place-, face-, word-, and body- selective voxels, the top-5 out of 10, 000 images from the IMRI stimulus set (NSD), and the top-5 images out of 000 total images as evaluated by the encoding component of BrainDiVE. For food selective voxels, the top-10 are visualized, A visual inspection indicates that our method is able to generate diverse images that semantically represent the target category. We further use CLIP to perform semantic probing of the images, and force the images to be classified into one of five categories. We measure the percentage of images that match the preferred category for a given set of voxels (Table 1). We find that our top-106 10% and 20% of images exceed the 1% and 2% of natural images in accuracy, indicating our method has high semantic specificity.
Faces
Places
Bodies
Words
Food
Mcan
S11
S2t
S1t
S2t
S1t
S21
SIt
S2t
S1t
S2t
S11
S2*
NSD all stim
17.4
17.2
29.9
29.5
31.6
31.8
10.3
10.6
10.8
10.9
20.0
20.0
NSD top-200
42.5
41.5
66.5
80.0
56.0
65.0
31.5
34.5
68.0
85.5
52.9
61.3
NSD top- 100
40.0
45.0
68.0
79.0
49.0
60.0
30.0
49.0
78.0
85.0
53.0
63.6
BrainDivE-22
69.5
70.0
97.5
100
75,5
68.5
60.0
57.5
89.0
94.0
78.3
75.8
BrainDiVE- 100
61.0
68.0
97.0
100
75.0
69.0
60.0
62.0
92.0
95.0
77.0
78.8
Table 1: Evaluating semantic specificity with zero-shot CLIP classification. We use CLIP to classify images Shown is the percentage where the classified category of the image matches the preferred category the brain region. We show this for each subject's entire NSD stimulus set (10, 000 images for $1 $1&2;, the top-200 and top-100 images (top-2% and top-1%) evaluated by mean true fMRI beta, and the top-200 and top- 100 (20% and 10%) of BrainDiVE images as self-evaluated by the encoding component of BrainDiVE. BrainDiVE generates images with higher semantic specificity than the top 1% of natural images for each brain region.
from each ROI into five semantic categories: hec//plac/bodd//wor//ooooo
5

---

Place voxels
Word vaxels
NSO (MAR))
BrainDivE
(Curs)
a
BIKE
HOU 2116 Body voxels
Face voxels
NSD (ANRI
BrainCWV (Curs)
Food voxels
NSD (MMR))
BrainDME
(Cuns)
Figure 4: Results for category selective voxels (S1). We identify the top-5 images from the stimulus set or generated by our method with highest average activation in each set of category selective voxels for the face/place/word/boo
categories, and the top-10 images for the food selective voxels.
Which ROI has more...
photorealistic faces
animals
abstract shapes/lines
S1
$2
$5
S7
S1
$2
$5
S7
S1
$2
$5
$7
FFA-NSD OFA-NSD
45
43
34
41
34
34
17
15
21
6
14
22
25
22
21
18
47
36
65
65
24
44
28
25
WFA-BrainDiVE
89
17
19
11
OFA-B BrainDiVE
4
15
22
71
61
52
50
80
79
40
39
Table Human evaluation of the difference between face-selective ROIs. Evaluators compare groups of images corresponding to OFA and FFA; comparisons are done within GT and generated images respectively. Questions are posed as: "Which group of images has more X?"; options are FFA/OFA/Same, Results are in %. Note that the "Same" responses not shown; responses across all three options sum 100.
4.3 Individual ROIS
In this section, we apply our method individual ROIs that are selective for the same broad semantic category. We focus on the occipital face area (OFA) and fusiform face area (FFA), as initial tests suggested little differentiation between ROIs within the place-, word-, and body- selective networks. In this experiment, we also compare our results against the top images for FFA and OFA from NeuroGen [9], using the top 100 out of 500 images provided by the authors. Following NeuroGen, we also generate 500 total images, targeting FFA and OFA separately (Figure 5). We observe that both
and NSD images have very high face content in FFA, whereas NeuroGen has higher animal face content. In OFA, we observe both NSD and BrainDiVE images have strong face component, although we also observe text selectivity in S2 and animal face selectivity in S5. Again NeuroGen predicts a higher animal component than face for S5. By avoiding the use of fixed categories, BrainDiVE images are more diverse than those of NeuroGen. This trend of face and animals appears at 2 and the much stricter t 5 threshold for identifying face- selective voxels (t > 5 used for visualization/evallation by BrainDiVE for FFA and OFA are consistent with past work suggesting that FFA represents faces at a higher level of abstraction than OFA, while OFA shows greater selectivity to low-level face features and sub-components, which could explain its activation by off-target categories [68, 69, 70].
The differences in images synthesized
To quantify these results, we perform a human study where subjects are asked to compare the top- 100 images between FFA & OFA, for both NSD and generated images. Results are shown in Table 2.

---

SS
NSD (14R0)
NeuroSen
(BOGAN)
BrainDME (Ours
BrainDME (Ours
OFA
55 OFA
NSO mARI
STOP
NeuroGen (BBGAN)
BrainVv
(Qurs (Dun 151
BainOVE
Figure 5: Results for face-selective ROIs For each ROI (OFA, FFA) we visualize the top-5 images from NSD and NeuroGen, and the top-10 from BrainDiVE. NSD images are selected using the fMRI betas averaged within each ROL. NeuroGen images are ranked according to their official predicted ROI activity means. BrainDiVE images are ranked using our predicted ROI activities from 500 images. Red outlines in the NSD images indicate examples of responsiveness to non-face content.
We find that OFA consistently has higher animal and abstract content than FFA. Most notably, this difference is on average more pronounced in the images from BrainDiVE, indicating that our approach is able to highlight subtle differences in semantic selectivity across regions.
COE S5 OPA
food
food 2
OPA
OPA 2
S1 food
S5 food
S1 OPA
Figure 6: Clustering within the food ROI and within OPA. Clustering of encoder model weights for each region is shown for two example subjects inflated cortical surface.
4.4 Semantic Divisions within ROIS
In this experiment, we investigate if our model can identify novel sub-divisions within existing ROIs. We first perform clustering on normalized per-voxel encoder weights using vmf-clustering [71]. We find consistent cosine difference between the cluster centers in the food-selective ROI well as in the occipital place area (OPA), clusters shown in Figure 6. In all four subjects, we observe relatively consistent anterior- pooeterrr split of OPA. While the clusters within the food ROI vary more anatomically, each subject appears to have more medial and a more lateral cluster. We visualize the images for the two food clusters Figure and the two OPA clusters in Figure 8. observe that for both the food ROI and OPA, the SeniDDiVE-ggeeraatt images from each cluster have noticeable differences in their visual and semantic properties. particular, the BrainDiVE images from food cluster-2 have much higher color saturation than those from cluster-1, 1, and also have more objects
7

---

Food cluster (fMRI)
Food cluster BiVE
cluster NSD (FMRT)
Food cluster BiVEnDiVE
S1
Figure 7: Comparing results across the food clusters. We visualize top-10 NSD fMRI (out of 10,000) and diffusion images (out of 500) for each cluster. While the first cluster largely consists of processed foods, the second cluster has more visible high color saturation foods, and more vegetables/fruit like objects. BrainDiVE helps highlight the differences between clusters.
Which cluster more
vegetables/fruits
healthy
colorful
away
S1
S2
S5
S7
S1
S2
S5
S7
S1
S2
S5
S7
S1
S2
S5
S7
Food-1 NSD
17
21
27
36
28
22
29
40
19
18
13
27
32
24
23
28
Food-2 NSD
65
56
56
49
50
47
54
45
42
52
53
42
34
39
36
42
Food-1 BrainDiVE
11
10
8
11
15
16
20
17
18
18
Food-2 BrainDiVE
80
75
67
64
68
68
46
51
79
82
65
difference between food clusters. Evaluators compare groups of images corresponding to food cluster (Food-1) 1) and food cluster 2 (Food-2), with questions posed as "Which group of images has/is more X?". Comparisons are done within NSD and generated images respectively. Note that the "Same" responses not shown; responses across all three options
Table Human evaluation
100. Results in %.
that resemble fruits and vegetables. In contrast, food cluster-1 generally lacks vegetables and mostly consist of bread-like foods. In OPA, cluster-1 is dominated by indoor scenes (rooms, hallways), while 2 is overwhelmingly outdoor scenes, with a mixture of natural and man-made structures viewed from far perspective. Some of these differences also present in the NSD images, but the differences appear to be highlighted in the generated images.
To confirm these effects, we perform human study (Table 3, Table 4) comparing the images from different clusters each ROI, for both NSD and generated images. expected from visual inspection of the images, we find that food cluster-2 is evaluated to have higher vegetable/fruit content, judged to be healthier, more colorful, and slightly more distant than food cluster-1. We find that OPA cluster-1 1 is evaluated to be more angular/gcometricc include more indoor scenes, to be less natural and consisting of less distant scenes. Again, while these trends are present in the NSD images, they more pronounced with the BrainDiVE images. This not only suggests that method has uncovered differences in semantic selectivity within pre-existing ROIs, but also reinforces the ability of BrainDiVE to identify and highlight core functional differences across visual cortex regions.

---

OPA cluster NSD (fMRI)
OPA cluster BrainDivE
$2
S7
OPA cluster 2 NSD (fMRI)
OPA cluster 2 BrainDivE
S2
S7
Figure 8: Comparing results across the OPA clusters. We visualize top-10 NSD fMRI (out of
10,000) and diffusion images (out of 500) for each cluster. While both consist of scene images, the first cluster have more indoor scenes, while the second has more outdoor scenes. The BrainDiVE images help highlight the differences in semantic properties.
Which cluster more...
angular/geometric
indoor
natural
far away
S7
S1
S2
S5
OPA-1 NSD
45
58
49
51
71
88
80
79
14
3
9
10
10
OPA-2 NSD
13
12
14
16
8
11
14
73
89
71
81
69
93
81
85
OPA-1 BrainDiVE
76
87
88
76
89
90
90
85
6
6
9
OPA-2 BrainDiVE
12
3
10
5
8
91
91
83
90
97
92
91
88
Table 4: Human evaluation of the difference between OPA clusters. Evaluators compare groups of images corresponding OPA cluster 1 (OPA-1) and OPA cluster 2 (OPA-2), with questions posed "Which group images more X?". Comparisons are done within NSD and generated images respectively. Note that "Same" responses are not shown; responses across all three options sum to 100. Results are in %.
Discussion
Limitations and Future Work Here, we show that BrainDiVE generates diverse and realistic images that can probe the human visual pathway. This approach relies on existing large datasets of natural images paired with brain recordings. In that the evaluation of synthesized images is necessarily qualitative, it will be important to validate whether our generated images and candidate features derived from these images indeed maximize responses their respective brain areas. As such, future work should involve the collection of human fMRI recordings using both our synthesized images and more focused stimuli designed to test our qualitative observations. Future work may also explore the images generated when BrainDiVE is applied to additional sub-region, new ROIs, or mixtures of ROIs.
Conclusion We introduce novel method for guiding diffusion models using brain activations BrainDiVE enabling to leverage generative models trained internet-scale image datasets for

---

data driven explorations of the brain. This allows to better characterize fine ineinnrined preferences across the visual system. We demonstrate that BrainDiVE can accurately capture the semantic selectivity of existing characterized regions. We further show that BrainDiVE can capture subtle differences between ROIs within the face selective network. Finally, we identify and highlight fine- grained subdivisions within existing food and place ROIs, differing in their selectivity for mid-level image features and semantic scene content. We validate our conclusions with extensive human evaluation of the images.
Acknowledgements
This work used Bridges-2 Pittsburgh Supercomputing Center through allocation SOC220017 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296. We also thank the Carnegie Mellon University Neuroscience Institute for support.
References
[1] Kalanit Grill-Spector and Rafael Malach. The human visual cortex. Annual Review of Neuro- science, 27:649-677, 2004. ISSN 0147006X. doi: 10. 1ANAANNURENNNUUUO0.00000 144220.
[2] J Sergent, S Ohta, and B MacDonald. Functional neuroanatomy of face and object processing: positron emission tomography study. Brain, 115:15-36, 1992.
[3] Nancy Kanwisher, Josh McDermott, and Marvin M Chun. The fusiform face area: a module human extrastriate cortex specialized for face perception. Journal of neuroscience, 17(11): 4302-49 4312,431,,,0 1997.
[4] Russell Epstein and Nancy Kanwisher. cortical representation the local visual environment. Nature, 392(6676):598-660 1998.
[5] Nidhi Jain, Aria Wang, Margaret M. Henderson, Ruogu Lin, Jacob S. Prince, Michael J. Tarr, and Leila Wehbe. Selectivity for food in human ventral visual cortex. Communications Biology 2023 6:1, 6:1-14, 2 2023. ISSN 2399-3642. doi: 10. 244.3346/
[6] Ian M L Pennock, Chris Racey, Emily J Allen, Yihan Wu, Thomas Naselaris, Kendrick N Kay, Anna Franklin, and Jenny Bosten. biased regions in the ventral visual pathway are food selective. Curr. Biol., 33(1):134- 146.e4, 2023.
[7] Meenakshi Khosla, N. Apurva Ratan Murty, and Nancy Kanwisher. A highly selective response to food in human visual cortex revealed by hypothesis-fiee free voxel decomposition. Current 2022.
Biology,
[8] A Ishai, L G Ungerleider, A Martin, L Schouten, and J V Haxby. Distributed representation of objects in the human ventral visual pathway. Proc Natl Acad Sei U S A, 96(16):93799923) 1999.
[9] Zijin Gu, Keith Wakefield Jamison, Meenakshi Khosla, Emily Allen, Yihan Wu, Ghislain St- Yve, Thomas Naselaris, Kendrick Kay, Mert R Sabuncu, and Amy Kuceyeski. NeuroGen: activation optimized image synthesis for discovery neuroscience. Neurolmage, 247:1 18812, 2022.
[10] Carlos Ponce, Will Xiao, Peter Schade, Till S Hartmann, Gabriel Kreiman, and Margaret Livingstone. Evolving images for visual neurons using deep generative network reveals coding principles and neuronal preferences. Cell, 177(4)999-1009, 2019.
[11] N Apurva Ratan Murty, Pouya Bashivan, Alex Abate, James DiCarlo, and Naney Kanwisher. Computational models of category brain regions enable high-throughput tests of selectivity. Nature communications, 12(1):5540, 2021.
10

---

[12] Nadine Chang, John A Pyles, Austin Mareus, Abhinav Gupta, Michael Tarr, and Elissa M Aminoff. Bold5000, a public fMRI dataset while viewing 5000 visual images. Scientific Data, 6(1):1-18, 2019.
[13] Emily J Allen, Ghislain St-Yves, Yihan Wu, Jesse L Breedlove, Jacob S Prince, Logan T Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, et al. A massive 7t fmri dataset bridge cognitive neuroscience and artificial intelligence. Nature neuroscience, 2022.
[14] Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:21 10741, 2021.
[15] Robert Desimone, Thomas D Albright, Charles G Gross, and Charles Bruce. Stimulus-selective properties of inferior temporal neurons in the macaque. Journal of Neuroscience, 4(8):2051- 2062, 1984.
[16] Laurent Cohen, Stanislas Dehaene, Lionel Naccache, Stéphane Lehéricy, Ghislaine Dehaene- Lambertz, Marie-Anne Hénaff, and François Michel. The visual word form area: spatial and temporal characterization of an initial stage of reading in normal subjects and posterior split-brain patients. Brain, 123(2);211-377, 2000.
[17] Paul E Downing, Yuhong Jiang, Miles Shuman, and Nancy Kanwisher. A cortical area selective for visual processing of the human body. Science, 293(5539):2470-2444 2001.
[18] Bruce D McCandliss, Laurent Cohen, and Stanislas Dehaene. The visual word form area: expertise for reading in the fusiform gyrus. Trends in cognitive sciences, 7(7):293-299, 2003.
[19] Meenakshi Khosla, N Apurva Ratan Murty, and Nancy Kanwisher. A highly selective response food in human visual cortex revealed by hypothesis-frct voxel decomposition. Current Biology, 32(19):4159-47 2022.
[20] Ian ML Pennock, Chris Racey, Emily J Allen, Yihan Wu, Thomas Naselaris, Kendrick N Kay, Anna Franklin, and Jenny M Bosten. Color-biased regions in the ventral visual pathway are food selective. Current Biology, 33(1):134 146, 2023.
[21] Jack L Gallant, Charles E Connor, and David Van Essen. Neural activity in areas vl, v2 and v4 during free viewing of natural scenes compared to controlled viewing. Neuroreport, 9(7): 1673-1678, 1678, 1998.
[22] Thomas Naselaris, Kendrick Kay, Shinji Nishimoto, and Jack Gallant. Encoding and decoding in fmri Neuroimage,56(()eee 410, 2011.
[23] Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan Solomon, Darren Seibert, and James DiCarlo. Performance-opumiin hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, 11 111(23):8619 8819)866188888 8624, 2014.
[24] Seyed-Mahdi Khaligh- Razavi and Nikolaus Kriegeskorte. Deep supervised. but not unsuper- vised, models may explain it cortical representation. PLoS computational biology, 10(11): e1003915, 2014.
Shaligh-Razavi
[25] Michael Eickenberg, Alexandre Gramfort, Gaël Varoquaux, and Bertrand Thirion. Seeing it all: Convolutional network layers map the function of the human visual system, Neurolmage, 152: 184-194, 2017.
[26] Haiguang Wen, Junxing Shi, Wei Chen, and Zhongming Liu. Deep residual network predicts cortical representation and organization of visual features for rapid categorization. Scientific reports, 8(1):3752, 2018.
[27] Jonas Kubilius, Martin Schrimpf, Kohitij Kar, Rishi Rajalingham, Ha Hong, Najib Majaj, Elias Issa, Pouya Bashivan, Jonathan Prescott-Roy. Kailyn Schmidt, et al. Brain-like object recognition with nigh-peeforming shallow recurrent anns. Advances in neural information processing systems, 32, 2019.

---

[28] Colin Conwell, Jacob S Prince, George Alvarez, and Talia Konkle. Large-scar scale benchmarking of diverse artificial vision models in prediction of 7t human neuroimaging data. bioRxiv, pages 2022-03, 2022.
[29] Tom Dupré la Tour, Michael Eickenberg, Anwar O Nunez-Elizalde and Jack L Gallant. Feature- space selection with banded ridge regression. Neurolmage, 264:119728 2022.
[30] Aria Yuan Wang. Kendrick Kay, Thomas Naselaris, Michael Tarr, and Leila Wehbe. Incor- porating natural language into vision models improves prediction and understanding of higher visual cortex. BioRxiv, pages 2022 6022-09.22 2022.
[31] Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXi:: 14, 2013.
[32] Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems, 30, 2017.
[33] Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM, 63(1 1):13-1144, 2020.
[34] Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International conference on machine learning, pages 1530-1538. PMLR, 2015.
[35] Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv: 14 10.85166, 16, 2014.
[36] Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research, 6(4), 2005.
[37] Jaseha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsuper- vised learning using nonequilibrium thermodynamics. International Conference on Machine Learning, pages 2265-2265. PMLR, 2015.
(38) Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:201 1.3466 2020.
[39] Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. Mach. Learn. Res., 23(47):1-33, 2022.
[40] Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint 141i 1784, 2014.
[41] Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv: /096,, 2018
[42] Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 2022.
[43] Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In International conference on machine learning, pages 1060-1069. PMLR, 2016.
[44] Robin Rombach, Andreas Blatmaann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High- resolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10684-10695, 2022.
[45] Zijiao Chen, Jiaxin Qing, Tiange Xiang, Wan Lin Yue, and Juan Helen Zhou. Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding. arXiv preprint arXiv:2211.000 1.66956, 1(2):4, 2022.
[46] Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. bioRxiv, pages 2022-11, 2022.
12

---

[47] Furkan Ozcelik and Rufin VanRullen. Brain-diffuser: Natural scene reconstruction from fmri signals using generative latent diffusion. arXiv preprint arXiv:2303.053334 2023.
[48] Yizhuo Lu, Changde Du, Dianpeng Wang, and Huiguang He. Minddiffuser: Controlled image reconstruction from human brain activity with semantic and structural diffusion. arXiv preprint arXiv:2303. 14139, 2023.
[49] Reese Kneeland, Jordyn Ojeda, Ghislain St-Yves, and Thomas Naselaris. Second sight: Using brain-optimized encoding models to align image distributions with human brain activity. ArXiv, 2023.
[50] Edgar Walker, Fabian Sinz, Erick Cobos, Taliah Muhammad, Emmanouil Froudarakis, Paul G Fahey, Alexander S Ecker, Jacob Reimer, Xaq Pitkow, and Andreas S Tolias. Inception loops discover what excites neurons most using deep predictive models. Nature neuroscience, 22 22(12):2000-2665 2019.
[51] Pouya Bashivan, Kohitij Kar, and James J DiCarlo. Neural population control via deep image synthesis. Science, (e34):443a, 2019.
[52] Meenakshi Khosla and Leila Wehbe. High-level visual areas act like domain-general filters with strong selectivity and functional specialization. bioRxiv, pages 2022-03, 2022.
[53] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion- 5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402, 2022.
[54] Pawel A Pierzchlewicz, Konstantin Willeke, Arne F Fix, Pavithra Elumalai, Kelli Restivo, Tori Shinn, Cate Nealley, Gabrielle Rodriguez, Saumil Patel, Katrin Franke, et al. Energy guided diffusion for generating neurally exciting images. bioRxiv, 2023.
[55] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[56] Aria Y. Wang, Ruogu Lin, Michael J. Tarr, and Leila Wehbe, Joint interpretation of represen- tations in neural network and the brain. In 'How Can Findings About The Brain Improve Al Systems?' Workshop @ ICLR 2021, 2021.
[57] Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303. 15389, 2023.
[58] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances Neural Information Processing Systems, 34:8780-8794, 2021.
[59] Wei Li, Xue Xu, Xinyan Xiao, Jiachen Liu, Hu Yang, Guohao Li, Zhanpeng Wang, Zhifan Feng, Qiaoqiao She, Yajuan Lyu, al. Upainting: Unified text-to-image diffusion generation with cross-modal guidance. arXiv preprint arXiv:2210. 1603/, 2022.
[60] Jacob S Prince, Ian Charest, Jan W Kurzawski, John A Pyles, Michael J Tarr, and Kendrick N Kay. Improving the accuracy of single-trial fmri response estimates using gimsingle. eLife, 11: c77599, nov 2022. ISSN 2050-084X. doi: 10. 10.7554/cLife.7755
[61] Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpm- solver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095, 2022.
[62] Susung Hong, Gyuseong Lee, Wooseok Jang, and Seungryong Kim. Improving sample quality of diffusion models using self-attention guidance. arXiv preprint arXiv:2210. 10.009200.00093, 2022.
[63] Alec Radford, Jong Wook Kim, Chris Hallacy A. Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In ICMI., 2021.
13

---

[64] Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917, 2022.
[65] Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmar- czyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In shitt Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2022.
[66]
Gabriel Iharco,, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong. John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021.
[67] A. Stigliani, K. S. Weiner, and K. Grill-Spector. Temporal processing capacity in high-level visual cortex domain specific. Journal of Neuroscience, 35:12412-12424, 2015. ISSN 0270-6474. doi: 10. (011523/NEUROOSC14453
14.2015.
[68] Jia Liu, Alison Harris, and Nancy Kanwisher. Perception of face parts and face configurations: An finri study. Journal of cognitive neuroscience, 22:203, 2010. ISSN 0898929X. doi: 10.1 N0.1162/JCCN2009222220
[69] David Pitcher, Vincent Walsh, and Bradley Duchaine. The role of the occipital face area in the cortical face perception network. Experimental brain research, 609:481-493 4 2011. ISSN 1432-1106. doi: 10. 10.1007/500221-01-2279999
[70] Maria Tsantani, Nikolaus Kriegeskorte, Katherine Storts, Adrian Lloyd Williams, Carolyn McGettigan, and Lúcia Garrido. Ffa and ofa encode distinct types of face identity information. Journal of Neuroscience, 41:1952 1969, 2021. ISSN 0270-6474. doi: 10. NEUUOOSI 24.9-20.2020.
[71] Arindam Banerjee, Inderjit S Dhillon, Joydeep Ghosh, Suvrit Sra, and Greg Ridgeway. Cluster- ing on the unit hypersphere using von mises-fisher distributions. Journal of Machine Learning Research, 6(9), 2005.
[72] Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Image synthesis and editing with stochastic differential equations. arXiv preprins arXiv:2/08.01073, 2021.
[73] Jun Hao Liew, Hanshu Yan, Daquan Zhou, and Jiashi Feng. MagicMix: Semantic mixing with diffusion models, 2022.
[74] Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.1 / 1929, 2020.
[75] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[76] Uya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint 1rXi:: 2017.
[77] James S Gao, Alexander G Huth, Mark D Lescroart, and Jack L Gallant. Pycortex: an interactive surface visualizer for fmri. Frontiers neuroinformatics, page 23, 2015.
14

---

Supplementary Material: Brain Diffusion for Visual Exploration
1. Broader Impacts (section
2. Visualization of each subject's category selective voxels (section 8)
3. CLIP zero-shot classification results for all subjects (section 9)
4. Image gradients and synthesis process (section 10)
5. Standard error for human behavioral studies (section 11)
6. Brain Encoder R (section 12)
7. Additional OFA and FFA visualizations (section 13)
8. Additional OPA and food clustering visualizations (section 14)
9. Training, inference, and experiment details (section 15)
7
Broader impacts
Our work introduces a method where brain responses as measured by IMRI can be used to guide diffusion models for image synthesis (BrainDiVE). We applied BrainDiVE to probe the representation of high-level semantic information in the human visual cortex. BrainDiVE relies on pretrained looole--fffsiion--222
and will necessarily reflect the biases in the data used to train these models. However, given the size and diversity of this training data, BrainDiVE may reveal data-driven principles of cortical organization that are unlikely have been identified using more constrained, hypothesis-driven experiments. As such, our work advances our current understanding of the human visual cortex and, with larger and more sensitive neuroscience datasets, may be utilized to facilitate future fine-grained discoveries regarding neural coding, which can then be validated using hypothesis-driis driven experiments.
15

---

Visualization of each subject's category selective voxel images
Place Voxels
Word Voxels
LAITAOS
BrainDiVE
a
BED
0O0
Face Voxels
Body Voxels
NSD
BrainDive (Gurs)
Food Voxels
BrainDive (Ours)
Figure S.1: Results for category selective voxels (S1). We identify the top-5 images from the stimulus or generated by our method with highest average activation in each set of category selective voxels for the face/place/word/boyy categories, and the top- 10 images for the food selective voxels, Note the top NSD body voxel image for SI was omitted from the main paper due content.
16

---

Place Voxels
Word Voxels
NSO
STOP
STOP
BrainD
FATIUS
BOYB
Face Voxels
Body Voxels
ASD
BrainDive (Durs)
Food Voxels
NED
BrainDiVE
Figure S.2: Results for category selective voxels (S2). We identify the top-5 images from the stimulus set or generated by our method with highest average activation in each set of category selective voxels for the face/place/word/bolt categories, and the top-10 images for the food selective voxels.
Place Voxels
Word Voxels
NEVES STOP LOPIE
NSD (MMRR))
FRAEI
BrainDivE (Qurs)
BATARR YEANUTEA YOA
KICKn!
Face Voxels
Body Voxels
ASD
BrainDiVE
(Ours]
Food Voxels
NSD
BrainDivE (Ours)
Figure S.3: Results for category selective voxels (S3). We identify the top-5 images from the stimulus set or generated by our method with highest average activation in each set of category selective voxels for the face/place/word/body categories, and the top-10 images for the food selective voxels.
17

---

Place Voxels
Word Voxels
NSD
STOP
BrainDIVE
SEPCEUST
End the
Face Voxels
Body Voxels
NSD
BrainDivE
Food Voxels
NSO
BrainDivE (Durs)
SLL
Figure S.4: Results for category selective voxels (S4). We identify the top-5 images from the stimulus set or generated by our method with highest average activation in each set of category selective voxels for the facc/place/word/body categories, and the top-10 images for the food selective voxels.
Place Voxels
Word Voxels
NSD
APIFR
BrainDive (Ours)
Swo
CAJU
NEERY
Face Voxels
Body Voxels
BrainDiVE
Food Voxels
NSD
BrainDivE (Ours)
Figure S.5: Results for category selective voxels (S5). We identify the top-5 images from the stimulus set or generated by our method with highest average activation in each set of category selective voxels for the face/place/word/body categories, and the top-10 images for the food selective voxels.
18

---

Place Voxels
Word Voxels
NED
BrainDiVE
Garfee
Face Voxels
Body Voxels
NSD
BrainDiVE
Food Voxels
NSD
BrainDivE (Durs)
EL
Figure S.6: Results for category selective voxels (S6). We identify the top-5 images from the stimulus set or generated by our method with highest average activation in each set of category selective voxels for the facc/place/word/body categories, and the top-10 images for the food selective voxels.
Place Voxels
Word Voxels
NSO (TMRU
STOP
BrainDivE
Face Voxels
Body Voxels
BrainDiVE
Food Voxels
NSD
BrainDive (Ours)
Figure S.7: Results for category selective voxels (S7). We identify the top-5 images from the stimulus set or generated by our method with highest average activation in each set of category selective voxels for the face/place/word/body categories, and the top-10 images for the food selective voxels.
19

---

Place Voxels
Word Voxels
TUND
STOP
BrainDive
Billy
ALLSA A
CANEL
Face Voxels
Body Voxels
NSD
BrainD
Food Voxels
NSD
BrainDive (Ours)
ESHHTE
Figure S.8: Results for category selective voxels (S8). We identify the top-5 images from the stimulus set or generated by our method with highest average activation in each set of category selective voxels for the face/place/word/body categories, and the top-10 images for the food selective voxels.
20

---

9 CLIP zero-shot classification
In this section we show the CLIP classification results for S1 S8, where Table S. in this Supple- mentary material matches that of Table in the main paper. We use CLIP [63] to classify images from each ROI into five semantic categories: face/place/hodyy//orr//ooo
Shown the percentage where the classified category of the image matches the preferred category of the brain region. We show this for the top-200 and top-100 images (top-2% and top-1%) 1)) evaluated by mean true fMRI beta, and the top-200 and top-100 (20% and 10%) of BrainDiVE images as self-evaluated by the encoding component of BrainDiVE. Please see Supplementary Section 15 for the prompts we use for CLIP classification.
Faces
Places
Bodies
Words
Food
Mcan
SI1
S21
Sit
S2
SIt
S21
SIt
S2
SIt
S22
SI
S21
NSD top-200
42.5
41.5
66.5
80.0
56.0
65.0
31.5
34.5
68.0
85.5
52.9
61.3
NSD top-100 100
40.0
45.0
68.0
79.0
49.0
60.0
30.0
49.0
78.0
85.0
53.0
63.6
BrainDiV
69.5
70.0
97.5
100
75.5
68.5
60.0
57.5
89.0
94.0
78.3
75.8
BrainDiVE-100
61.0
68.0
97.0
100
75.0
69.0
60.0
62.0
92.0
95.0
77.0
78.8
Table S.1: Evaluating semantic specificity with zero-shot CLIP classification for S1 and S2
Faces
Places
Bodies
Words
Food
Mean
S31
S41
S3t
S41
S3t
S4
S31
S41
S31
S41
S31
S4
NSD top-200
33.0
39.0
74.5
71.5
57.9
47.5
27.0
20.5
49.5
$3.5
48.4
46.4
NSD top- 100
38.0
41.0
81.0
72.0
60.0
49.0
30.0
25.0
46.0
57.9
51.0
49.0
BrainDiVE
67.5
73.5
99.0
100
59.0
66.5
61.0
31.0
85.0
89.0
74.3
72.0
BrainDiVE-100
67.0
71.0
100
100
59.0
72.0
61.0
34.0
89.0
93.0
75.2
74.0
Table S.2: Evaluating semantic specificity with zero-shot CLIP classification for S3 and S4
Faces
Places
Bodies
Words
Food
Mean
S5t
S6t
S51
S6t
S5
S61
S51
S6t
S51
S6t
S5
S6
NSD top-200
41.0
38.5
89.5
56.9
57.9
56.5
33.5
34,0
77.0
55.5
59.8
48.3
NSD top- 100
45.0
46.0
93.0
55.0
54,0
61.0
33.0
32.0
85.0
56.9
62.0
50.2
BrainDiVE
67.0
63.0
99.5
96.0
74.0
66.0
75.0
68.0
83.5
79.0
79.8
74.4
BrainDiVE-
64.0
57.9
100
99.0
77.0
72.0
80.0
75.0
87.0
83.0
81.6
77.4
Table S.3: Evaluating semantic specificity with zero-shot CLIP classification for S5 and S6
Faces
Places
Bodies
Words
Food
Mean
S71
S8t
S7t
S8t
S7t
S81
S71
S8t
S71
S8t
S7t
SS
NSD top-200
38.5
34.0
71.0
57.5
61.0
56.5
20.5
24.5
52.0
36.5
48.6
41.8
NSD top- 100
35.0
36.0
76.0
48.0
63.0
61.0
26.0
21.0
56.0
37.0
51.2
40.6
HrainDiVE--200
73.0
77.5
93.5
94.5
65.0
64.5
31.0
56.5
85.5
55.5
69.6
69.7
BrainDiVE-
69.0
72.0
94.0
94.0
65.0
67.0
25.0
56.0
92.0
74.0
69.0
72.6
Table S.4: Evaluating semantic specificity with zero-shot CLIP classification for S7 and S8.
21

---

10 Image gradients and synthesis process
In this section, we show examples of the image at each step of the synthesis process. We perform this visualization for face-, place,, body-, word-, and food- selective voxels. Two visualizations are shown for each set of voxels, we use S1 for all visualizations in this section. The diffusion model is guided only by the objective of maximizing a given set of voxels. We observe that coarse image structure emerges very early on from brain guidance. Furthermore, the gradient and diffusion model sometimes work against each other. For example in Figure S.14 for body voxels, the brain gradient induces the addition of an extra arm, while the diffusion has already generated three natural bodies. Or in Figure S.15 for word voxels, where the brain gradient attempts to add horizontal words, but they are warped by the diffusion model. Future work could explore early guidance only, as described in "SDEdit" and "MagicMix" [72, 73].
10.1 Face voxels
We show examples where the end result contains multiple faces (Figure S.9), or a single face (Figure 1.1)).
t=1...10
t=11...20
t=21...30
t=31...40 40
t=41..50 50
Figure S.9: Example I of face voxel guided image synthesis for SI. We utilize 50 steps of Multistep DPM-Solver+++. We visualize the gradient magnitude w.r.t. the latent (top, normalized at each step for visualization) and the weighted euler RGB image that the brain encoder accepts (bottom).
22

---

t=1...10
t=11..200
t=21...30 30
t=31..40 40
t=41..50 50
Figure S.10: Example 2 of face voxel guided image synthesis for S1. We utilize 50 steps of Multistep DPM-Solver++. We visualize the gradient magnitude w.r.t. the latent (top, normalized at each step for visualization) and the weighted euler RGB image that the brain encoder accepts (bottom)
23

---

10.2
Place voxels
We show examples where the end result contains an indoor scene (Figure S.11), or an outdoor scene (Figure S.12).
t1...10
t=11..20
t=21...30
t=31..440 .0.
t=41..50 .0
Figure S.11: Example of place voxel guided image synthesis for S1. We utilize 50 steps of Multistep DPM-Solver++. We visualize the gradient magnitude w.r.t. the latent (top, normalized each step for visualization) and the weighted euler RGB image that the brain encoder accepts (bottom).
24

---

t=...00
t=11..20
t=21...30 30
.0
t=31...40
t=41..50 50
Figure S.12: Example of place voxel guided image synthesis for S1. We utilize 50 steps of Multistep DPM-Solver++. We visualize the gradient magnitude w.r.t. the latent (top, normalized at each step for visualization) and the weighted euler RGB image that the brain encoder accepts (bottom).
25

---

10.3
Body voxels
We show examples where the end result contains an single person's 's body (Figure S.13), or an multiple people (Figure S.14).
t=1..10
t=11...00
t=21...30
t=31...40 .0
t=41..50 .0
Figure S.13: Example 1 of body voxel guided image synthesis for S1. We utilize 50 steps of Multistep DPM--Sller++. We visualize the gradient magnitude w.r.t. the latent (top, normalized each step for visualization) and the weighted euler RGB image that the brain encoder accepts (bottom).
26

---

t=1..10
t=11..20
t=21...30 30
t=31..40
t=41...00 50
Figure S.14: Example of body voxel guided image synthesis for S1. We utilize 50 steps of Multistep DPM-Solver++.. We visualize the gradient magnitude w.r.t. the latent (top, normalized at each step for visualization) and the weighted euler RGB image that the brain encoder accepts (bottom).
27

---

10.4
Word voxels
We show examples where the end result contains recognizable words (Figure S.15), or glyph like objects (Figure S.16).
t=1..10
RES
SAME
t=11..20
CO
DOCKS
SUDDS
HILL CAENS
GALE CAEKS
SOLE TOLES
GOLE LAEYS
BLE PRESS
GOL LAESS
NEW
BLEINES
NUTIES
NIRISS
TERTES
RES
t=21...30
BULEAEYS
OULACSS
BOLEAAYS OS
BOLEAAYS OF 13
SOLD EAENS
BRIEAAKSS
BALE OA
RALACS
M RS5
PUT 1S3
NAT RS
M WE5
OF
t=31...40
LALELAESS 1.7155
GLLLASSS LATES
RILEAAS
BALE
BALE
TQUE LAELS
BRLELAESS
RELLAESS RA
BOLE
LALS
RELAESS
RATESS
RE
NUT
t=41...00 .0
GULAES UAS
ROLLASS
BOLLAESS MF
REDEAS KS
QULAES MSS
ROLAES NS
RALLEYS M'
RULAES MA
RELLES
IS
GF
Figure S.15: Example of word voxel guided image synthesis for S1. We utilize 50 steps of Multistep DPM-Solver++. We visualize the gradient magnitude w.r.t. the latent (top, normalized each step for visualization) and the weighted euler RGB image that the brain encoder accepts (bottom).
28

---

t=1...10
t=11..20
g
30
t=21...30
CO
c
co
co
Eg
co
co
Eg
Eg
co
t=31..40 40
co
Eg
cg
cg
Eg
co
cg
Eg
Eg
t=41..50 .0
Eg
E
Eg
Egl
Egl
Eg
Fggl F
Fg
Figure S.16: Example of word voxel guided image synthesis for S1. We utilize 50 steps of
Multistep DPM-Solver++. We visualize the gradient magnitude w.r.t. the latent (top, normalized
at each step for visualization) and the weighted euler RGB image that the brain encoder accepts
(bottom).
29

---

10.5 Food voxels
We show examples where the end result contains highly processed foods (Figure S.17, showing what appears to be a cake), or cooked food containing vegetables (Figure S. 1.18).
t=1...10
t=11...0
t=21...30
t=31...40 .00
t=41..50 .0
Figure S.17: Example 1 of food voxel guided image synthesis for S1. We utilize 50 steps of Multistep DPM-Solver++. We visualize the gradient magnitude w.r.t. the latent (top, normalized each step for visualization) and the weighted euler RGB image that the brain encoder accepts (bottom).
30

---

t=1...0
t=11..200
t=21..330 30
t=31..40
t=41..50 50
Figure S.18: Example of food voxel guided image synthesis for S1. We utilize 50 steps of Multistep +oM--Slver++..... We visualize the gradient magnitude w.r.t. the latent (top, normalized at each step for visualization) and the weighted euler RGB image that the brain encoder accepts (bottom).
31

---

11 Human behavioral study standard error
In this section, we show the human behavioral study results along with the standard error of the responses. Each question was answered by exactly 10 subjects from prolific. co. In each table, the results are show in the following format: Mean(SEM). Where Mean is the average response, while SEM is the standard error of the mean ratio across 10 subjects: (SEM
V10
Which ROI has more...
photorealistic faces
animals
abstract shupes/lines
S1
S2
S5
S7
51
52
55
57
51
52
S5
S7
FFA-NSD
45(7.2)
43/8.3)
34.6.2
41(6.5)
34(1.5)
34(3.5)
17(4.0)
15(3.8)
21(6.8)
6(1,0)
14(2.9)
22(6.6)
OFA-NSD
25(5.1)
22(6.4)
21(5.6)
18(5.3)
47(3.2)
36.2.5
65(5.7)
65(6.4)
24(8.5)
44(9.2)
28(8.1)
25(6.4)
FFA- BrainiiDiiV
7947.8)
8941.)
6055.3)
52(5.3)
17(5.6)
13(3.5)
21(3.9)
19(2.2)
633.21
11(6.4)
184.9)
20X6.6)
OFA BrViDiriDDiVE
(16.7)
4(2.5)
15(2.)
22(5.1)
71(8.4)
61(8.2)
52(5.1)
50(3.5
80(5.8)
79(7.4)
40/5.8
39(7.1
Table S.5: Human evaluation of the difference between face-selective ROIs. Evaluators compare groups of images corresponding to OFA and FFA; comparisons are done within GT and generated images respectively. Questions are posed as: "Which group of images has more X?": options are FFA/OFA/Same. Results are in %.
Note that the "Same" responses are not shown; responses across all three options sum to 100.
Which cluster is more...
vegetables/fruits
healthy
S5
S7
S1
S2
S7
Food-1 NSD
17(4.3)
21((.8)
27(5.1)
36(3.5)
28(5.8)
22(3.7)
29(6.2)
40(4.0)
Food-2 NSD
65(7.2)
56(6.4)
56(5.7)
49(3.9)
50(7.1)
47(4.9)
54(6.0)
45(4.3)
Food-1 BrainDiVE
11(7.0)
10(6.0)
8(6.6)
11(6.5)
15(6.2)
16(6.0)
20(7.2)
17(7.1)
Food-2 BrainDiVE
80(73)
75(8.0)
67(9.8)
64(7.4)
68(7.7)
68(7.)
46(9.3)
51(7.8)
Which cluster is more...
colorful
far away
S1
$2
S5
S7
S1
$2
S5
S7
Food-1 NSD
19(5.5)
18(6.1)
13(2.8)
27(3.5)
32(6.6)
24(4.7)
23(6.5)
28(4.2)
Food-2 NSD
42(6.4)
52(5.6)
53(6.5)
42(6.4)
34(7.0)
19(8.))
36(7.9)
42(7.3)
Food-1 BrainDiVE
6(3.8)
9(5.7)
11(5.7)
16(4.9)
24(6.8)
18(6.4)
27(8.9)
18(6.0)
Food-2 BrainDiVE
79(7.9)
82(6.9)
65(7.6)
61(8.9)
39(10.)
51(9.0)
39(8.8)
40(8.8)
Table S.6: Human evaluation of the difference between food clusters. Evaluators compare groups of images corresponding to food cluster 1 (Food-1) 1) and food cluster 2 (Food-2), with questions posed as "Which group of images has/is more X?". Comparisons are done within NSD and generated images respectively. Note that the "Same" responses are not shown; responses across all three options sum to 100. Results are in %.
Which cluster is more...
angularfgenmetrric
indoor
S1
S2
S5
S7
S1
S2
S5
S7
OPA-1 NSD
45(7.2)
58(9.4)
49(7.7)
51(9.0)
71(5.6)
88(4.9)
80(5.1)
79(5.6)
OPA-2 NSD
13(4.0)
12(2.4)
14(2.9)
16(4.5)
7(3.2)
8(3.7)
11(3.0)
14(4.3)
OPA-1 Biai
76(7.8)
87(.6)
88(6.6)
76(7.8)
89(5.6)
90(5.7)
90(4.7)
85(5.3)
OPA-2 BrainDiVE
12(4.9)
3(2.0)
4(1.5)
10(4.2)
7(3.2)
7(3.2)
5(2.1)
8(2.4)
Which cluster is more...
natural
lar away
S1
S2
S5
S7
$1
$2
S5
$7
OPA-T NSD
14(3.8)
3(2.0)
9(411)
10(2.8)
10(2.4)
((0.9)
6(2.9)
8(2.4)
OPA-2 NSD
73(3.4)
89(7.4)
71(6.4)
81(6.1)
69(4.6)
93(3.8)
81(6.5)
85(5.5)
OPA-I BrainDiVE
6(3.2)
6(1.5)
9(3.6)
6(2.9)
1(0.9)
3(2.8)
3(2.8)
8(5.6)
OPA-2 BrainDiVE
91(5.7)
91(3.6)
83(6.9)
90(5.5)
97(2.8)
92(6.6)
91(5.6)
88(7.4)
Table S.7: Human evaluation of the difference between OPA clusters. Evaluators compare groups of images
corresponding to OPA cluster I 1)) and OPA cluster 2 (OPA-2), with questions posed as "Which group of images is more X?". Comparisons are done within NSD and generated images respectively. Note that the "Same" responses are not shown; responses across all three options sum to 100. Results are in %.
32

---

12
Brain encoder R
S1
S2
S3
S4
S5
S6
S7
S8
Figure S.19: Visualization of R2 on test set images. We evaluate R² the 1000 images shared by all subjects. Note that voxels in early visual or outside of higher visual are not modeled.
In Figure 12 we show the R² of the brain encoder as evaluated on the test images. Our brain encoder consists of CLIP backbone and linear adaptation layer. We do model voxels in the early visual cortex, nor do we model voxels outside of higher visual. Our model can generally achieve high R² in regions in known regions of visual semantic selectivity.
33

---

13 OFA and FFA visualizations
In this section, we visualize the top-10 NSD and BrainDiVE images for OFA and FFA. NSD images are selected using the fMRI betas averaged within each ROI. BrainDiVE images are ranked using our predicted ROI activities from 500 images.
S1 FFA
BrainDIVE
S1 OFA
NSO
BrainDIvE
Figure S.20: Results for face-selective ROIs in S1.
$2 FFA
BrainDIve (Durs)
S2 OFA
NSO
STOP Caee
STOP
BrivEDIVE (Ours)
CP
AIILLIAN
DEVE
Figure S.21: Results for face-selective ROIs in S2.
34

---

S5 FFA
NSO
BrainDivE (Gurs)
OFA
NSD
BrainDiVE
Figure S.22: Results for face-selective ROIs S5.
S7 FFA
NSD
GREEN
BrainDiVE
OFA
NSD
BrainDive
Figure S.23: Results for face-selective ROIs S7.
35

---

14
OPA and food visualizations
food
food 2
S1 food
S2 food
55 food
S7 food
BODI
OPA
OPA
S1 OPA
S2 OPA
S5 OPA
S7 OPA
Figure S.24: Clustering within the food ROI and within OPA. Clustering of encoder model weights for each region shown for four subjects on an inflated cortical surface.
Consistent with Jain al. [5], we observe that food voxels themselves are anatomically variable across subjects, while the two food clusters form alternating patches within the food patches. OPA generally yields anatomically consistent clusters in the four subjects we investigated, with four subjects showing anterior-posterios split for OPA.
OPA cluster NSD (fMRI)
cluster BiainDiVE
S1
$2
OPA cluster 2 NSD (fMRI)
OPA cluster 2 BrainDivE
$2
Figure S.25: Comparing results across the OPA clusters for S1 and S2.
36

---

OPA cluster 1 NSD (fMRI)
OPA cluster 1 BrainDivE
S5
$7
OPA cluster NSD (fMRI)
OPA cluster 2 BrainDiVE
SS
S7
Figure S.26: Comparing results across the OPA clusters for S5 and S7.
37

---

Food cluster NSD (tMI
Food cluster BrainDiVE
S1
$2
Food cluster NSD (MRRI
Food cluster BrainDivE
Figure S.27: Comparing results across the food clusters for S1 and S2.
38

---

Food cluster NSD (tMRI)
Food cluster 1 BrainDIVE
S7
EE
Food cluster 2 NSD (MRRI
Food cluster 2 BrainDivE
Figure S.28: Comparing results across the food clusters for S5 and S7.
39

---

15 Training, inference, and compute details
Encoder training. Our encoder backbone laion2b_ _e____3eb______ provided by OpenCLIP [65, 66]. The ViT [74] weights for the brain encoder frozen. We train linear layer consisting of weight and bias to map from the 512 dimensional vector to higher visual voxels B. The CLIP image branch outputs are normalized to the unit sphere.
ViT-B/16 with CLIP pretrained weights
CLIP ing (T)
MI W x [CLIP ing (I) +
Training is done using the Adam optimizer [75] with learning rate irinit 3e 4 and 1r and 1.5e - 4, with learning rate adjusting exponentially each epoch. We train for 100 epochs. Decoupled weight decay [76] of magnitude decay 2e is applied. Each subject is trained independently using the ~ 9000 images unique to each subject's stimulus set, with 122 evaluated on the ~ 1000 images shared by all subjects.
During training of the encoder weights, the image is resized to 224 x 224 to match the input size of ViT-B/16. We augment the images by first randomly scaling the pixels by value between [0.95, 1.05], then normalize the image using OpenCLIP VIT image mean and variance. Prior to input to the network, further randomly offset the image spatially by up pixels along the height and width dimensions. The empty pixels are filled in using edge value padding. A small amount of gaussian noise (00 0.05?) is added to each pixel prior input to the encoder backbone.
Objective. For all experiments, the objective used is the maximization of selected of voxels. Here we will further draw a link between the optimization objective we use and the traditional CLIP text prompt guidance objective [14, 59]. Recall that Ma is our brain activation encoder that maps from the image to per-voxel activations. It accepts as input an image, passes it through a ViT backbone, normalizes that vector to the unit sphere, then applies a linear mapping to go to per-voxel activations. S E N are the set of voxels we are currently trying to maximize (where N is the set of all voxels in the brain), ? is a step size parameter, and Dn the decoder from the latent diffusion model that outputs an RGB image (we ignore the euler approximation for clarity). Also recall that we use diffusion model that performs c-prediction.
In the general case, we perturb the denoising process by trying to maximize a set of voxels S:
Ceheta
Ctheta
vI - V
|S|
Σ
(c(((()())
iS
For the purpose of this section, will focus on a single voxel first, then discuss the moxti-voxel objective.
In our case, the single voxel perturbation is (assuming W is a vector, and that (.-) the inner product):
éhheta Etheta
VI aj V.. (7Mo
VI a; V., 0. (Igen)
= Etheta
CLIP ing (Ieca)
Etheta - 7VI--a (IW,
) + 6)
CCLIP sng (Ieen) ||
We can ignore b, as it does not affect optimal CLIP ing (Igen)
CLIP Lng (Igen)
Etheta 7VI at ((W,
||LIPI (Igen) ll2
Now let us consider the typical CLIP guidance objective for diffusion models, where Prext is the guidance prompt, and CLIP is the text encoder component of CLIP:
CLIP text (Prest
CLIP ing (Igen)
Cshata TV1 a. Vn.
|LIP text (Peex)]l2 CLIP ing (Igen)
As such, the W that we find by linearly fitting CLIP image embeddings to brain activation plays the role of text prompt. In reality, ||]]|2 * 1 (but norm is constant for each voxel), and there is likely
40

---

no computationally efficient way to "invert" W directly into human interpretable text prompt. By performing brain guidance, we are essentially using the diffusion model synthesize an image Zgen where in addition to satisfying the natural image constraint, the image also attempts to satisfy:
CLIP ing (Igen) |CII ing (Zg
W
||
Or put another way, it generates images where the CLIP latent aligned with the direction of W. Let now consider the multi-voxel perturbation, where W,, by is the per-voxel weight vector and bias:
v1 at
Mz(Dn(z()).)
Ethnta Etheta
S
iES
y [S]
We move
outside of the gradient operation
Etheta
7
V. 7 VI a+ V
VI a4 V
Mo (oD((z))))
[S]
CLIP ing (Iyen)
= Ctheta
[S]
(W,
CLLIP: ing (gea) 12
+ bi
E
We again ignore bj as it does not affect gradient
7 VI as V
CLIP ing (Igen) (W
= Etheta
S
CLLIP ing (Iean) 112
iES
We can move
Σ outside due to the distributive nature of gradients
7
CLIP iin (Igen) CLIP smg (Igen) 112
Etheta
C4
xx (W,,
[
Thus from a gradient perspective, the total gradient is the average of gradients from all voxels. Recall that the inner product is bilinear function, and that the CLIP image latent is on the unit sphere. Then we are generating an image that
CLIP ing (Igen)
ELES W Will2
CLIP ing (Ige)]l2
1Eies
Where the optimal image has CLIP latent that is aligned with the direction of Eies W..
Compute. We perform our experiments on a cluster of Nvidia V100 GPUs in either 16GB or 32GB VRAM configuration, and all experiments consumed approximately 1, 500 compute hours. Each image takes between 20 and 30 seconds to synthesize. All experiments were performed using PyTorch, with cortex visualizations done using PyCortex [77].
CLIP prompts. Here we list the text prompts that are used to classify the images for Table 1. in the main paper.
face_class ["A face facing the camera", "A photo of face", "A photo of human face", "A photo of faces", "A photo of a person's 's face", "A person looking the camera", "People looking the camera", "A portrait of person", "A portrait photo"]
body_class = ["A photo of a torso", "A photo of torsos", "A photo of limbs", "A photo of bodies", "A photo of a person", photo of people"]
scene scene_class class ["A photo a bedroom", "A photo of an office", "A photo of hallway", "A photo of doorway", "A photo of interior design", "A photo
41

---

of building", "A photo of house", "A photo of nature", "A photo of landscape", landscape photo", "A photo of trees", "A photo of grass"]
flod_class ["A photo of food"]
text_clase ["A photo of words", "A photo of glyphs", "A photo of glyph", "A photo of text", "A photo of numbers", "A photo of a letter", "A photo of letters", "A photo of writing", "A photo of text on an object"]
We classify an image as belonging to a category if the image's CLIP latent has highest cosine similarity with the CLIP latent of prompt belonging to given category. The same prompts are used to classify the NSD and generated images.
42


Reviews:
[{"review_id": "n7mth2zYYb", "reviewer": "Reviewer_MC3H", "summary": "This paper introduces Brain Diffusion for Visual Exploration (BrainDiVE) that aimed at exploring the fine-grained functional organization of the human visual cortex. Motivated by the limitations of previous studies that relied on researcher-crafted stimuli, BrainDiVE leverages generative deep learning models trained on large-scale image datasets and brain activation data from fMRI recordings. The proposed method uses brain maps as guidance to synthesize diverse and realistic images, enabling data-driven exploration of semantic preferences across visual cortical regions. By applying BrainDiVE to category-selective voxels and individual ROIs, the authors demonstrate its ability to capture semantic selectivity and identify subtle differences in response properties within specific brain networks. It is also shown that BrainDiVE identifies novel functional subdivisions within existing ROIs, highlighting its potential for providing new insights into the human visual system's functional properties.", "questions": "1. Validation of BrainDiVE's Effectiveness:\n  a. Can the authors provide more details on the validation process to ensure that BrainDiVE-generated images indeed effectively activate the targeted brain regions? \n  b. Could the authors consider conducting more extensive comparisons with other existing methods [e.g. 9 and 10], to demonstrate the advantages and uniqueness of BrainDiVE in eliciting specific brain activations?\n  c. Could the authors show the results on decoding a baseline region which is not involved in visual processing?\n\n2. Clarity in Methodology: The methodology part could benefit from more clarity and detailed explanations of key components, such as the diffusion model architecture and the exact implementation of brain-guided synthesis. Improving the architecture in Figure-2 might help.\n\n3. Statistical Analysis of Qualitative Evaluation: As the paper relies on qualitative evaluation with 10 subjects, could the authors mention this explicitly and what limitations are expected from this in the limitations section?\n\n4. Ethical Considerations: Could the authors describe the demographics of the population, or acknowledge the lack of this information to inform of wether the results are biased to a specific gender or population group.\n\n5. Reproducibility: Are the authors intending to release the code and not publicly available data such as the scores produced by the human raters upon acceptance of the paper? This is key to guarantee reproducibility", "limitations": "Authors have not addressed the negative societal impact of their work, but this can be fixed by adding specific text in the Discussion section. The authors don't mention the demographic characteristics of the small human subject database that they have used not mention any ethical concerns related to decoding images from brain activations. This is fixable so authors should address it.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper's methodology shows promise in capturing semantic selectivity and identifying fine-grained functional distinctions within visual cortical regions. Also, the paper's potential significance lies in applying BrainDiVE to understand the fine-grained functional organisation of the human visual cortex. \nBy providing insights into category selectivity, response properties, and sub-regional divisions, the paper opens avenues for further exploratory neuroscience studies. \nTo achieve this, the authors perform the experiments in the manuscript are extensive, covering:\n1. the semantic specificity of the method by decoding images from task fMRI and literature-obtained ROIs\n2. compared the abstraction of face representation in the brain by comparing images decoded from two regions, the fusiform face area (FFA) and the occipital face area (OFA). \n3. Use the method to extend knowledge of brain function by finding subdivisions in known areas in the cases of food decoding.\nThe authors describe the experimental setting clearly at the beginning of section 4 for all cases.\nThe experiments show that the manuscript has a good balance between the combination of known methodological approaches and addressing interesting questions in neuroscience. Specifically, the authors perform a commendable effort in obtaining quantitative results using human evaluators (cf Tables 3 and 4).\nFinally, the authors show through results evidence that their method is a window into understanding region-specificity hypotheses of brain regions which are knowingly involved in different aspects of visual processing.", "weaknesses": "The paper could benefit from comparing its results with previous contributions [e.g. 9 and 10] to highlight its novelty and contributions in relation to existing methods. \nThe methodology part lacks clarity, particularly in explaining key components like the diffusion model architecture and brain-guided synthesis—further clarifying the role of the image-to-brain encoder in influencing the denoising process during inference.\nThe generalisability and reliability of results are hard to asses through a small dataset, specifically just 10 subjects. \nFurthermore, the paper lacks clarity on how the sub-divisions of the visual cortex are being verified or validated. Whether the results are specific to the analysed regions or the algorithm is being too biased by the experimental condition is not clear through the experiments. For instance, what would happen if the authors try to decode an area not specific to visual processing? An example of this would be using subsections of the orbitofrontal gyrus or other brain areas not expected to perform well in reconstructing visual stimuli.\n\nAs a small point authors should review the presentation of images. Figure 1 shows mostly best-case scenarios which are then not as good in Figure 4 (for instance the case of images generated from face voxels). This might bias readers. Second, some details, such as figure 4 appearing before figure 3 make reading the paper confusing.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the excellent suggestions. We address specific questions below, and will include additional details in the general response.\n> **Formatting and presentation**\n\nWe will update our paper to follow other image synthesis papers like OpenAI's DALL-E 2 [1] and GigaGAN [2], they similarly select/curate images for the first figure and note this in the caption. We will also update the figure numbering.\n\n> **Q1.1) Validation of effectiveness**\n\nFor within ROI clusters, OPA & food clusters remained stable across random cluster initializations, and showed the highest cosine distance for voxel embeddings among all ROIs we examine. \n\nMore broadly, we agree on the importance of validating and conducting fMRI studies to evaluate BrainDiVE. **We have ongoing work in this area**. \n\nIn the paper we perform validation via two methods:\n1. When we target voxels from widely accepted category selective regions (faces, places, bodies, food, word), we perform CLIP 5-way classification using a natural language probe. The probe sets follow the guidelines detailed in section 3.1.4 of [4]. We provide our full probe set in page 26 of our supplemental. \n\n    We found that images created using BrainDiVE were aligned with scientific consensus for broad category selective visual regions. \n\n2. When voxels from single ROIs (FFA/OFA), and within ROIs (Food/OPA) are targeted, we perform behavioral studies. Ten different subjects were recruited for each region (FFA, OFA, Food, OPA) via Prolific. This yielded 80 evaluations per-subject per-question, for 10 subjects each. We find that BrainDiVE can highlight differences between ROIs (FFA vs OPA) and within ROIs (Color in food voxels, and indoor/outdoor in OPA). This is important as it may facilitate future open-ended data driven exploration of the visual cortex. \n\nWe further validate using an alternative fMRI encoder backbone in the **response PDF**, and find our images can achieve high predicted activations.\n> **Q1.2) Comparisons with existing methods**\n\nWe concur that our work builds upon Gu et al. and Ponce et al. ([9, 10] in text). In particular, the setup of our work is similar to [9], as we both use the NSD dataset and operate offline (no live subject). Our work is not directly comparable to [10] as their experiments use real-time macaque brain feedback, which our work and [9] do not have.\n\nUsing BrainDiVE, we probe the visual cortex selectivity of the brain at three hierarchical stages, and offer new insight on region wise selectivity. Briefly, our work differs from [9] in two important ways:\n\n1. We use a diffusion model trained on billions of internet images, three magnitudes larger than BigGAN (ImageNet), this is important as we are not restricted to single object images that dominate ImageNet. We further leverage the model's training with classifier-free guidance [5] (∇log[p(y|x;t)] = ∇log[p(x|y;t)]-∇log[p(x;t)], with y=text, and x=image) to use the model in an unconditional mode, and combine it with our brain encoder to enable brain conditioned sampling. This is in contrast to Gu et al., where they use BigGAN trained on imageNet and condition on class-labels. \n\n2. The brain optimized image generation process is very different. [9] uses search then optimize. They first sample images the 1000 ImageNet classes, and **non-differentiably** select the top-10 classes for each region. Finally they perform brain gradient updates of the GAN latent. This results in images with less diversity, and many of their images are nearly identical. In contrast, our work yields diverse images by using end-to-end differentiable optimization, **and do not restrict the search space to a fixed image category a priori**.\n\nPlease see the **response PDF** for more comparisons.\n\n> **Q1.3) Non-visual cortex results**\n\nPlease see **response PDF** for OFC results. As expected, we find our method does not yield consistent semantics in OFC.\n\n> **Q2) Clarity in methodology**\n\nWe agree that this could be further clarified. We will provide an updated Figure 2 in an upcoming revision.\n\nThe brain encoder predicts fMRI activations from images. Diffusion models predict the score (derivative of log-likelihood) of the image distribution, and can be treated as a special class of energy-based models. A brain encoder that outputs real-numbers can be interpreted as an energy function [6], the derivative of which can be additively combined with the diffusion to enable conditional sampling of naturalistic images that maximize brain response.\n\n> **Q3) Statistical analysis of human evaluation**\n\nEach of the 10 subject performed 80 binary evaluations per question, we collect 8800 total responses (11 questions, 10 splits, 4 NSD subjects, for both BrainDiVE/real images). Due to space constraints, we provided standard error (SE) measurements in section 5 of the supplemental. We will provide further statistical analysis in an upcoming revision.\n\n> **Q4) Ethical Considerations**\n\nFor 30 subjects, they are between age 22~63, 14 women/15 men/1 unknown, 28 white, 2 black, 1 mixed, 1 unknown. We currently have a \"Broader Impacts\" section in the supplemental, we will update this in the revision to discuss the demographics and additional issues.\n\n> **Q5) Code release**\n\nYes, we will release code and human evaluation data upon acceptance. We have also sent the AC a comment linking to an anonymous repo containing our code.\n\n⠀\n\nWe're grateful for your clear and helpful suggestions. In light of our response, we hope you might view our work in a more positive light. Please feel free to let us know if you have additional comments! \n\n⠀\n\n[1] Hierarchical Text-Conditional Image Generation with CLIP Latents\n\n[2] Scaling up gans for text-to-image synthesis\n\n[3] Selectivity for food in human ventral visual cortex\n\n[4] Learning transferable visual models from natural language supervision\n\n[5] Classifier-free diffusion guidance\n \n[6] Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and mcmc"}, {"review_id": "PIUS8yf4Pa", "reviewer": "Reviewer_fpHd", "summary": "This study proposes BrainDIVE, a system for synthesizing optimal stimuli for any given region of interest in the brain. The model combines a pretrained latent diffusion model for image generation with a linear “brain encoder” trained to map CLIP feature vectors onto the corresponding brain activity. At test time, a gradient-based optimization iteratively produces an image that maximizes the activity of a predefined set of voxels. The technique is validated on well-known ROIs, producing the expected images. It can also highlight subtle differences between ROIs selective to the same broad category, or functional subdivisions of existing ROIs. The functional hypotheses are validated by human subjects evaluating the properties of the generated pictures. The technique appears simple but can provide meaningful information for neuroscience studies.", "questions": "* I do not understand why you systematically need to report the top-5 or top-10 out of 500 generated images? Do I understand correctly that the pool of 500 are all generated for the same objective (ROI maximization)? If yes, then what does this imply about the remaining 99% of images? Are they worse than NeuroGen? Are they actually failure cases? What do the worse 1% of images look like? If they are not representative of the expected category, does that invalidate your optimization method?\n* Appendix, section 9, training objective: this section is useful to understand the image optimization process. One thing I do not understand from this section or from Figure 2 is why the optimization is performed in the diffusion latent space, rather than in CLIP space? Why do the gradients have to flow all the way into the diffusion model? Couldn’t you iteratively optimize the $CLIP_{img}$ vector for the same objective, and then use it to condition the diffusion model in one pass (as you would with a text prompt)? This should be computationally more efficient, and I don’t see any reason why it would be less accurate. \n", "limitations": "Limitations are properly acknowledged.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* The method can retrieve subtle differences between ROIs selective for the same class (e.g. OFA and FFA).\n* The method can highlight functional differences between sub-clusters of existing ROIS (e.g. food clusters)\n* The qualitative observations are validated by behavioral evaluations from human subjects. \n", "weaknesses": "* The Related works section tends to overstate the novelty of the technique. It only mentions NeuroGen as prior work, whereas other prior studies had also attempted to generate optimal stimuli for specific ROIs: for instance, Ratan Murty et al (2021), Ozcelik et al (2022), Ozcelik et al (2023). The latter also used a diffusion model for image generation, as done here. Furthermore, you seem to be aware of at least some of these studies, since you criticized one of them in your Methods section for using a “hand-derived” prior. It would be much better to properly acknowledge all these studies in the Related works section, and clarify your method’s advantages/disadvantages (e.g. no hand-derived prior/time-consuming iterative method).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your detailed and concrete suggestions! We will incorporate all of your feedback into our paper.\n\n> **Comparisons to prior work**\n\nIndeed, our work builds upon the foundations laid by Ratan Murty et al. (2021),  Ozcelik et al. (2021), and Ozcelik et al. (2023). It was not our intention to imply otherwise. \n\nWe originally included two of these papers in the \"Related Works\" sections of our paper (Murty on Line 35 and 72, Ozcelik on Line 67). In a revised version, we will explicitly highlight the connections between our work and these referenced papers and will additionally cite Ozcelik et al. (2021). \n\nBriefly:\n1. Ratan Murty et al. (2021) builds upon an adversarial trained BigGAN image generator trained on imageNet with category conditioning. Their brain encoder consists of an ImageNet trained frozen ResNet50 backbone, along with a linear decoder layer. \n\n    Our work utilizes a text-conditioned diffusion image model trained on score matching with classifier-free guidance [1] (∇log[p(y|x;t)] = ∇log[p(x|y;t)]-∇log[p(x;t)]; y=text, x=image). This enables our diffusion model to sample from the unconditional image distribution via score function ∇log[p(x;t)] in the absence of text conditioning. Our brain encoder consist of a frozen ViT-B/16 backbone along with a linear probe, which enables brain conditioned image synthesis.\n    \n    A major difference lies in the image synthesis process. As BigGAN is trained on category conditioning (1000 classes in a one-hot vector), Murty et al. perform a joint optimization over c (category) and z (latent). As the category is discrete, they use softmax(c\\) to facilitate gradient optimization. It is not clear that a softmax is suitable, as BigGAN is not trained with non-discrete classes. An alternative could be gumbel-softmax [1], but this has high gradient variance.\n    \n    In contrast, our work performs end-to-end differentiable optimization. So BrainDiVE is not restricted to a particular ImageNet class like NeuroGen by Gu et al., or convex combination of class embeddings as in Murty et al.\n    \n2. The method by Ozcelik et al. (2023) is state of the art for brain based image reconstruction using diffusion models, and is conditioned on a pattern of brain activations. \n\n    In contrast, the primary goal of our work is to generate images that are predicted to maximize a given region of the brain, and not reconstruction. In BrainDiVE, we only need to train an fMRI encoder, and the region can be flexibly defined as any subset of the brain without retraining.\n\n   In Ozcelik et al., when they perform region activation, they set activated voxels to 1, and others to 0. This is followed by manual latent normalization. They achieve intriguing results using this hand-crafted heuristic, but have the implicit assumption that other voxels are zeroed when one region is active.\n\n> **Q1): On top-5 or top-10**\n\nTo clarify, the top-5 and top-10 are for qualitative **visualization only**. Our numerical evaluation follows prior work in image generative models (DALL-E [2]) and brain based activating image synthesis (NeuroGen by Gu et al. 2022, citation 9 in paper), and uses top-20% like NeuroGen unless otherwise noted.\n\nOur top-5/top-10 for visualization is done automatically using the brain encoder, without any manual cherry picking. \n\nFor numerical results, we use the brain encoder without manual cherry picking to evaluate the top 10% and 20% (100 or 200 images) in 4.2, and follow NeuroGen and use the top 20% in 4.3/4.4. Notably this is used for visualization *and* evaluation in OpenAI's DALL-E (called reranking) `all samples used for both qualitative and quantitative results ... use reranking with N = 512` [2] where they use `top 32 of 512 after reranking with CLIP, but we do not use any manual cherry-picking` (online post). Like DALL-E, reranking is used in NeuroGen (Gu et al. 2022) where they only analyze the top-100 of 500 images as reranked using their brain encoder (confirmed with author), and only visualize the top-k of 500 images. We also rerank using our brain encoder.\n\nIn general, diffusion models do not always converge during generation. The bottom 1% for the brain generally look like the bottom 1% for text-conditioned, and usually have no recognizable objects. This is an active area of research (see Imagen's dynamic thresholding approach).\n\n> **Q2): Optimizing CLIP latent**\n\nThis is indeed an idea we had as well! We believe there are broadly two ideas you are touching upon:\n\n1. Given that we know the optimal fixed CLIP image embedding (512 or 768 dim), in theory it should be easy to condition the diffusion model on this embedding to replace the typical CLIP text embedding. \n\n    In practice high performance diffusion models are text-conditioned, but are not conditioned on a fixed sized (512/768) text embedding, and instead are conditioned on 75 token embeddings (plus start/end tokens) for an entire sentence in order to model visual semantic composition. \n\n    Solving for a fixed sized CLIP image embedding does not give us the full text embedding. In addition, there exists a modality gap between CLIP image and text latents [3], and closing the gap is an active area of research.\n    \n2. An alternative may be to solve for the CLIP text embedding directly via gradient descent. However current work [4] requires a handcrafted text prompt, and has been demonstrated only for single objects. In addition, these approaches require ground truth images for a given concept, which we do not have.\n\n⠀\n\nThank you again for your wonderful suggestions, and we're eager to hear your thoughts on our clarifications. Please let us know if you have any additional questions or comments!\n\n⠀\n\n[1] Categorical Reparameterization with Gumbel-Softmax\n\n[2] Zero-Shot Text-to-Image Generation\n\n[3] Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning\n\n[4] An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion"}, {"review_id": "7ve0qfLUO7", "reviewer": "Reviewer_SV7e", "summary": "The paper presents a new algorithm to guide a diffusion model to decode maximally activating images for particular voxel subregions of human fMRI using an encoding model trained to predict brain activity from images. The algorithm identifies stereotypic features for defined ROIs, such as faces or food items. The authors evaluate the specificity of the reconstructed image with CLIP zero-shot classification and human evaluation. They also cluster the encoding model weights and find that the clusters result in perceivable semantic categories in the reconstructed images. This algorithm allows for more fine-grained analysis of preferences across the visual system. ", "questions": "Q1: As far as I understand $\\gamma$ trades off between the denoising gradient and the maximization gradient. I couldn't find what value you set it too, or how you chose it. \n\nQ2: There is a preprint that describes pretty much the same idea as in the paper but uses it for V4 single neurons:\n\nPawel A. Pierzchlewicz, Konstantin F. Willeke, Arne F. Nix, Pavithra Elumalai, Kelli Restivo, Tori Shinn, Cate Nealley, Gabrielle Rodriguez, Saumil Patel, Katrin Franke, Andreas S. Tolias, Fabian H. Sinz Energy Guided Diffusion for Generating Neurally Exciting Images\nhttps://www.biorxiv.org/content/10.1101/2023.05.18.541176v1\n\nIf one looks at the date, the preprint has been submitted around the NeurIPS deadline, so it's unlikely that the authors were aware of it and I would consider this as a case where two groups had the same idea. However, I think it would be fair to mention/cite them.  \n\nQ3: Can you discuss the compute time required for a single image?\n", "limitations": "Limitations are discussed. ", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "There have been many papers that reconstruct images from brain activity. However, all of them require some form of retraining the diffusion model. This approach only needs an encoding model that can be trained independently. \n\nThe extensive human evaluation is great. While this goes beyond what I would ask for a NeurIPS paper, I think it would be great to also verify them by showing them back to human subjects. ", "weaknesses": "The authors show that the reconstructed images possess stable properties that can be identified by humans and by a CLIP network. However, they don't show that the images also do what they are supposed to do, which is activate a particular subnetwork. While human experiments probably go beyond the scope of this NeurIPS paper, one way to do that would be to take **another** encoding model for the same neural activity, take it as a proxy for the brain and show the images to that. \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are deeply grateful for your excellent suggestions! We will address specific questions below. Please see the PDF in the general response for additional figures. \n\n> **Additional validation**\n\nWe agree with your comments on additional validation. **There is ongoing work to investigate the performance of BrainDiVE synthesized images in humans using fMRI.**\n\nFollowing your suggestion, we train an alternative encoding model using EVA02_CLIP_B_psz16_s8B [1], this is a very recently published backbone model with TrV blocks (in contrast to the ViT B/16 backbone with ViT blocks we use in the paper) jointly trained using mask image modeling and image-text contrastive loss (backbone we use in paper is image-text contrastive loss alone), which has been independently validated to achieve high ImageNet performance. We freeze the model weights, and train a linear probe with bias to estimate the fMRI activations. We validate that the new model can achieve high $R^2$ that is comparable to our current backbone. \n\n**Please see the PDF in the general response for a distribution plot of predicted activations.** We find that our BrainDiVE images can achieve high predicted neural activations when valdiated on a new backbone.\n\n> **Q1) Choice of $\\gamma$**\n\nIndeed, a high $\\gamma$ gives more weight to the brain activation gradient (more activating, less natural), while a low $\\gamma$ gives more weight to the diffusion gradient (more natural, less activating). We set $\\gamma$=130, and this is described in Line 149 of the original paper. There is a typo here and it was represented as $\\eta$, we will update the revision to clarify this value.\n\nWe performed search in increments of 10, exploring values between 10.0 up to values of 300.0, and synthesized 100 images for each of the broad category selective regions and recorded the gradient values at each step and the final synthesized images. We found that values between 110 and 150 yielded gradient magnitudes that largely matched between the brain and diffusion at early time-steps when coarse image structure emerged. The value of 130 was selected as a median value. This is the only hyperparameter that we introduce on top of diffusion models.\n\nApproaches like reflected diffusion [2] or dynamic thresholding from Imagen [3] may enable higher $\\gamma$ values, and remain an avenue for future research.\n\n> **Q2) On concurrent work**\n\nWe were not previously aware of the work by Pierzchlewicz et al. (2023) on \"Energy Guided Diffusion for Generating Neurally Exciting Images\" [4]. After closely reading the paper, we agree that the ideas in our paper and their paper are similar.\n\nBoth works tackle the synthesis of images that are predicted to activate regions of the brain. A broad difference is that our paper targets the higher visual cortex in humans which are believed to encode semantic concepts, while they target V4 in macaques.  This yields differences in the final images, where they primarily target the synthesis of complex visual patterns, while we primarily target the synthesis of compositional visual scenes. \n\nWe thank the reviewer for bringing this interesting paper to our attention, and will include it as a citation in our upcoming revision.\n\n> **Q3) Compute time**\n\nWhen using fp16 diffusion models, with fp32 brain encoders, 50 steps of denoising for images at 512x512 resolution requires around 25 seconds on a Nvidia V100 (from 2017). We briefly discuss this in section 9 of our supplemental, and will clarify this in the main paper in an upcoming revision.\n\nIn practice our experiments were performed on a cluster and consumed 1500 GPU compute hours. Accelerating this processing is a very interesting avenue of research, and we discuss approaches inspired by MagicMix/SDedit [5,6] in section 4 of our supplemental to speed up our work. \n\n⠀\n\nWe hope the clarifications have been insightful. Please don't hesitate to comment if there are any additional questions!\n\n⠀\n\n[1] EVA-CLIP: Improved Training Techniques for CLIP at Scale\n\n[2] Reflected Diffusion Models\n\n[3] Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding\n\n[4] Energy Guided Diffusion for Generating Neurally Exciting Images\n\n[5] MagicMix: Semantic Mixing with Diffusion Models\n\n[6] SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equations"}, {"review_id": "XLw4Ih6pgi", "reviewer": "Reviewer_Vg6A", "summary": "This paper proposes a diffusion-based method for fMRI-guided image synthesis and claims that it can identify the selectivity of various ROIs in the visual cortex. The motivation of this work is well-defined and is of vital importance to the computational neuroscience field. While the method in this work is intuitive and lacks machine learning depth.", "questions": "Please consider the things listed in the “Weaknesses” section.\n\nAlso please consider providing information regarding the interpretability of the utilization of the diffusion model in your context.", "limitations": "To my knowledge and understanding, there is no potential negative societal impact of this work. Other limitations please see “Weaknesses”.", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "Strengths: \n* Interesting scientific field and question of neuroscience.\n* The paper is well-written and has a smooth and concrete flow.\n* The experiments and visualizations are in good condition and sufficient.", "weaknesses": "Weaknesses\n\n* The method used in this paper is heuristic and lacks technical depth. \n    * Generating or reconstructing images from fMRI is not a new idea, as has been proposed in [1], etc.\n    * CLIP is an off-the-shelf large-scale pre-trained model, performing conditional diffusion-based generation with embedding from CLIP or Stable Diffusion is stereotyped [2,3].\n    * In Section 3.3, the domain-specific techniques you designed for this specific task are just some first-order tricks (i.e., linear combination of S, euler approximation). Maybe these tricks are effective enough, but that's overly empirical.\n* The experimental section is empirical and lacks theoretical analysis. Your analysis of the fMRI encoder with ROIs is interesting, but little focus had been put on what's the scientific relationship of neuroscience and diffusion model.\n\n\n\n[1] https://www.biorxiv.org/content/10.1101/2022.11.18.517004v3\n\n[2] https://arxiv.org/abs/2112.10752\n\n[3] https://arxiv.org/pdf/2208.01618.pdf", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are strongly encouraged by your evaluation that our work is interesting for neuroscience, it is well-written, and it has good experiments. We appreciate your advice on clarification. We address specific comments below and refer to the general response for results.\n\n> **Scope of the paper**\n\nWe would like to clarify that our work proposes to use diffusion models as a component to elucidate the functional role of different regions of the brain. Our work does not investigate the use of diffusion models for image reconstruction from fMRI or image generation from text. We will clarify this in the upcoming revision.\n\nWe are using diffusion models because they have been demonstrated to be state of the art estimators of the natural image prior [1]. These models have been trained on billions of images (the model we use is trained on LAION-2B), which is three magnitudes larger than BigGAN models trained on ImageNet. We further use the fact that diffusion models can be interpreted as a a score function (gradient of the log-likelihood), and can be combined with a derivative of an energy function (gradient of the brain w.r.t. input) [2].\n\n> **Q1.1) Comparison to Takagi et al.**\n\nThe paper by Takagi et al. [3] is relevant and interesting. It is a work we already cite (Line 67) in our own work. However the problem that we investigate is very different from Takagi et al.\n\nTakagi et al. investigates *reconstructing visual stimuli from a fixed set of voxels* in the brain. Our work explores the synthesis of **novel visual stimuli** that are predicted to activate parts of the visual cortex, not reconstruction of seen stimuli. The method proposed in our paper is intended as a tool for future data-driven investigations of the human visual cortex. \n\nAs our method is gradient based, it is permutation invariant [4] and can be flexibly applied to any subset of visual cortex voxels given a voxel-wise encoding model.\n\n**To our knowledge, our paper is the among the first to explore the synthesis of images predicted to activate higher visual cortex with diffusion models.** There is concurrent work [5] which explores using diffusion models to synthesize images predicted to activate macaque V4. Previous work on diffusion models with fMRI tackle reconstruction.\n\n> **Q1.2) On text-conditioned image synthesis**\n\nTo clarify, **we are not performing text-conditioned image synthesis** using CLIP embeddings as done in [6,7].\n\nOur use of CLIP is purely as a backbone for the visual encoder for fMRI. The use of an pretrained backbone with a linear decoding layer is common in voxel-wise encoding models for fMRI neuroscience, and CLIP models are the best backbone for higher visual cortex in fMRI [8].\n\nThe optimization method proposed in [7] requires knowledge of ground truth images for a given concept, which we do not have.\n\n> **Q1.3) Use of linear combination of S and euler estimation**\n\n* **On a linear combination of S**\n  * A linear combination of voxels (S) activations is accepted in neuroscience investigations for fMRI. In fMRI, voxels are often clustered in regions of interest (ROIs), where voxels have similar selectivity and behavior to visual stimuli. In ROIs, it is common to evaluate the average activations of all voxels using a linear combination (average of S). **This approach has been used in some of the most important papers for visual fMRI [9, 10].**\n\n* **On an euler estimation**\n  * The euler estimate is often used when the gradients of a network are used to guide image synthesis. Concurrent work [5], which uses diffusion models to generate images that are predicted to activate macaque V4 neurons, **also uses an euler estimate to estimate a clean sample that is fed to their encoder (see e.q. 6 in [5])**. This approach is acknowledged in other work ([11] section 2.3 last paragraph; And [12] e.q. 4). We adopt this technique since the euler estimate is in the distribution of natural images, and only use it to modify the image provided to the encoder, not the diffusion output directly.\n\n> **Q2) Theoretical analysis of our work**\n\nWe have included a theoretical analysis of our work in **section 9 of our supplemental.**\n\nIn that section, we show that with an encoder backbone of fixed norm, the brain maximization objective for a single/multiple voxels yields an image where the CLIP image embedding is equal to the normalized voxel weight.\n\n> **Interpretability**\n\nWe show in **section 4 of the supplemental** how the brain and the diffusion model work together. We find that low-level details often emerge first, and that the brain signal and diffusion signal are not always harmonious. \n\n⠀\n\nWe genuinely appreciate the reviewer's commitment to ensuring the rigor of our paper. Their insights have undeniably contributed to its refinement. The clarifications we've provided underscore the innovative aspects of our research and its potential contributions to the field. We respectfully invite the reviewer to approach our work with a more optimistic viewpoint. \n\n⠀\n\n[1] Diffusion Models Beat GANs on Image Synthesis\n\n[2] Reduce, reuse, recycle: Compositional generation with energy-based diffusion models and mcmc\n\n[3] High-resolution image reconstruction with latent diffusion models from human brain activity\n\n[4] Deep Set Prediction Networks\n\n[5] Energy Guided Diffusion for Generating Neurally Exciting Images\n\n[6] High-Resolution Image Synthesis with Latent Diffusion Models\n\n[7] An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversion\n\n[8] What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines?\n\n[9] The fusiform face area: a cortical region specialized for the perception of faces\n\n[10] A cortical representation of the local visual environment\n\n[11] GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models\n\n[12] UPainting: Unified Text-to-Image Diffusion Generation with Cross-modal Guidance"}]

Author rebuttal:
We are grateful to all reviewers for their constructive suggestions, which we agree will significantly improve the communication of our work.

We are very encouraged by reviewers’ evaluation on the quality of this paper. All four reviewers find the work interesting ("methodology shows promise in capturing semantic selectivity" (MC3H); "The method can retrieve subtle differences between ROIs" (fpHd); compared to prior works -- "this approach only needs an encoding model that can be trained independently" (SV7e); "The experiments and visualizations are in good condition and sufficient" (Vg6A)). 


### General clarifications
### 1. Scope and experiments

* We propose BrainDiVE -- a method that utilizes diffusion models to investigate functional specialization in the higher visual cortex. 
* Our work relies on end-to-end differentiable optimization with an image-computable voxel-wise fMRI encoder, and leverages an existing large-scale image diffusion model without retraining. It can be flexibly targeted at arbitrary regions in the visual cortex without retraining.
* We apply BrainDiVE to explore the brain's visual cortex selectivity at three hierarchical levels and offer new scientific insight on the selectivity of different regions. 
    * First, we apply it to widely accepted category selective regions -- faces, places, bodies, food, words.
    * Second, we apply it to ROIs that code for faces at different levels in the feature hierarchy (FFA, OFA). Our results are in line with the widely held belief that OFA responds to lower level face features relative to FFA. 
    * Third, we apply it to splits within food-selective and place-selective (OPA) ROIs, where we identify potentially novel functional subdivisions.
* Our evaluation is done in two different ways
    * For category selective regions, we perform CLIP 5-way classification using natural language. The design of our prompts follow the best practices defined in 3.1.4 of [1]. The prompts are available in section 9 of the supplemental. We observe that BrainDiVE images indeed capture the category specificity of these regions.
    * For FFA and OFA, which both code for face; subregions of food; and subregions of OPA which codes for scenes -- we perform a human behavioral study to validate the fine-grained visual-semantic attributes. We recruit 10 subjects for each set of comparisons. This results in a total of 8800 total responses (11 questions, 10 splits, 4 NSD subjects, for both BrainDiVE and real images). We report standard error metrics in section 5 of the supplemental. We find that BrainDiVE can highlight differences in preferred attributes between visual regions, suggesting that it can be useful for future data driven exploration of the visual cortex.

The ROI masks are derived from functional localizer results from the official NSD paper (faces, places, bodies, words, OFA, OPA), or obtained from other authors directly (food) [2].

We include visualizations of the image synthesis process and brain gradients in **section 4** of the supplemental. We perform a theoretical evaluation of our work in **section 9** of the supplemental. 

We agree in the importance of evaluating the images using real humans, and an effort to collect fMRI data is ongoing.

### 2. Relationship to concurrent and prior work

To our knowledge, we are the first to apply diffusion models to investigate the selectivity of the human higher visual cortex. Unlike prior work, we are not doing image reconstruction. There is concurrent work by Pierzchlewicz et al. [3] which applies diffusion models to investigate the selectivity of macaque V4, we will cite this in an upcoming revision.

In our work we cite papers which we believe are highly relevant (in-paper citation numbers): Gu et al. [9], Ponce et al. [10], Murty et al. [11], Takagi et al. [46], Ozcelik et al. [47]. We build upon the insights from these works. Our work is most similar to [9], as we use the same dataset, while [10] uses real-time responses of macaque visual cortex neurons for image synthesis.

* Unlike [9, 11], we perform end-to-end gradient optimization, and do not rely on fixed categories identified via search as in [9], or a softmax relaxation of categories as in [11]. In addition, we use a diffusion model trained on billions of images. Due to this, our synthesized images are not restricted to single object images that dominate the 1000 classes from ImageNet as in [9, 11].
* Different from [46, 47], we are not performing reconstruction of visual stimuli. Instead, we are proposing to use a diffusion model as a component for the synthesis of novel visual stimuli that is predicted to match the selectivity of a region. We do not rely on a fixed sized input, and regions can be flexibly defined without retraining. [47] achieves intriguing results by setting voxels to 0/1 followed by latent normalization, but this relies on the implicit assumption that voxels outside a region are inactive. 

We will include a more extensive discussion of related work in an upcoming revision. 

### 3. New figures/results in response PDF
1. We further validate the BrainDiVE results using an encoder with an alternative pretrained backbone. We find that the results are robust.
2. There is a more extensive comparison of BrainDiVE and NeuroGen from Gu et al. This highlights the semantic fidelity and diversity of our images.
3. We apply BrainDiVE to a region outside the visual cortex, which is not known to have visual selectivity. We confirm that the images do not show consistent semantic trends.

Additional results will be included in an upcoming revision.

⠀

We genuinely appreciate the suggestions, and believe our paper will be improved with your feedback. Please let us know if you have any additional questions or comments!

⠀

[1] Learning transferable visual models from natural language supervision

[2] Selectivity for food in human ventral visual cortex

[3] Energy Guided Diffusion for Generating Neurally Exciting Images

Meta-review: The paper presents a novel approach to reconstruct images from brain activity that only needs an encoding model that can be trained independently.
The main interest is that uses the most recent developments on generative models of vision to perform novel analyses of the visual cortex in humans and obtain non-trivial results on the semantic specialization of visual regions.
The method can indeed retrieve subtle differences between ROIs selective for the same class (e.g. OFA and FFA) and highlight functional differences between sub-clusters of existing ROIS.
The qualitative observations are validated by behavioral evaluations from human subjects.

The paper is well-written, and the experiments and visualizations are in good condition and sufficient.
On the weakness side, the technical contribution of the paper is overall less impressive.   

There is a wide consensus among reviewers, hence the paper is certainly well suited for NeurIPS, and certainly for a spotlight presentation.
Decision: Accept (oral)