Title: Brain Diffusion for Visual Exploration: Cortical Discovery using Large Scale Generative Models
Venue: NeurIPS 2023
Accepted: yes
Keywords: neuroscience

brain

fmri

generative models

diffusion models

image synthesis

visual cortex
Abstract: A long standing goal in neuroscience has been to elucidate the functional organization of the brain. Within higher visual cortex, functional accounts have remained relatively coarse, focusing on regions of interest (ROIs) and taking the form of selectivity for broad categories such as faces, places, bodies, food, or words. Because the identification of such ROIs has typically relied on manually assembled stimulus sets consisting of isolated objects in non-ecological contexts, exploring functional organization without robust a priori hypotheses has been challenging. To overcome these limitations, we introduce a data-driven approach in which we synthesize images predicted to activate a given brain region using paired natural images and fMRI recordings, bypassing the need for category-specific stimuli. Our approach -- Brain Diffusion for Visual Exploration ("BrainDiVE") -- builds on recent generative methods by combining large-scale diffusion models with brain-guided image synthesis. Validating our method, we demonstrate the ability to synthesize preferred images with appropriate semantic specificity for well-characterized category-selective ROIs. We then show that BrainDiVE can characterize differences between ROIs selective for the same high-level category. Finally we identify novel functional subdivisions within these ROIs, validated with behavioral data. These  results advance our understanding of the fine-grained functional organization of human visual cortex, and provide well-specified constraints for further examination of cortical organization using hypothesis-driven methods.

Brain Diffusion for Visual Exploration: Cortical Discovery using Large Scale Generative Models

Abstract

A long standing goal in neuroscience has been to elucidate the functional organization of the brain. Within higher visual cortex, functional accounts have remained relatively coarse, focusing on regions of interest (ROIs) and taking the form of selectivity for broad categories such as faces, places, bodies, food, or words. Because the identification of such ROIs has typically relied on manually assembled stimulus sets consisting of isolated objects in non-ecological contexts, exploring functional organization without robust a priori hypotheses has been challenging. To overcome these limitations, we introduce a data-driven approach in which we synthesize images predicted to activate a given brain region using paired natural images and fMRI recordings, bypassing the need for category-specific stimuli. Our approach - Brain Diffusion for Visual Exploration ('BrainDiVE') builds on recent generative methods by combining large-scale diffusion models with brain-guided image synthesis. Validating our method, we demonstrate the ability to synthesize preferred images with appropriate semantic specificity for well-characterized category-selective ROIs. We then show that BrainDiVE can characterize differences between ROIs selective for the same high-level category. Finally we identify novel functional subdivisions within these ROIs, validated with behavioral data. These results advance our understanding of the fine-grained functional organization of human visual cortex, and provide well-specified constraints for further examination of cortical organization using hypothesis-driven methods. Code and project site: https://www.cs.cmu.edu/~afluo/BrainDiVE

1 Introduction

The human visual cortex plays a fundamental role in our ability to process, interpret, and act on visual information. While previous studies have provided important evidence that regions in the higher visual cortex preferentially process complex semantic categories such as faces, places, bodies, words, and food [1, 2, 3, 4, 5, 6, 7], these important discoveries have been primarily achieved through the use of researcher-crafted stimuli. However, hand-selected, synthetic stimuli may bias the results or may not accurately capture the complexity and variability of natural scenes, sometimes leading to debates about the interpretation and validity of identified functional regions [8]. Furthermore, mapping selectivity based on responses to a fixed set of stimuli is necessarily limited, in that it can only identify selectivity for the stimulus properties that are sampled. For these reasons, data-driven methods for interpreting high-dimensional neural tuning are complementary to traditional approaches.

We introduce Brain Diffusion for Visual Exploration ('BrainDiVE'), a generative approach for synthesizing images that are predicted to activate a given region in the human visual cortex. Several

Place regions

Word regions

Face regions

FHQH

CPRM

cL

EL.

recent studies have yielded intriguing results by combining deep generative models with brain guidance [9, 10, 11]. BrainDiVE, enabled by the recent availability of large-scale fMRI datasets based on natural scene images [12, 13], allows us to further leverage state-of-the-art diffusion models in identifying fine-grained functional specialization in an objective and data-driven manner. BrainDiVE is based on image diffusion models which are typically driven by text prompts in order to generate synthetic stimuli [14]. We replace these prompts with maximization of voxels in given brain areas. The result being that the resultant synthesized images are tailored to targeted regions in higher-order visual areas. Analysis of these images enables data-driven exploration of the underlying feature preferences for different visual cortical sub-regions. Importantly, because the synthesized images are optimized to maximize the response of a given sub-region, these images emphasize and isolate critical feature preferences beyond what was present in the original stimulus images used in collecting the brain data. To validate our findings, we further performed several human behavioral studies that confirmed the semantic identities of our synthesized images.

More broadly, we establish that BrainDiVE can synthesize novel images (Figure 1) for categoryselective brain regions with high semantic specificity. Importantly, we further show that BrainDiVE can identify ROI-wise differences in selectivity that map to ecologically relevant properties. Building on this result, we are able to identify novel functional distinctions within sub-regions of existing ROIs. Such results demonstrate that BrainDiVE can be used in a data-driven manner to enable new insights into the fine-grained functional organization of the human visual cortex.

2 Related work

Mapping High-Level Selectivity in the Visual Cortex. Certain regions within the higher visual cortex are believed to specialize in distinct aspects of visual processing, such as the perception of faces, places, bodies, food, and words [15, 3, 4, 1, 16, 17, 18, 19, 5, 20]. Many of these discoveries rely on carefully handcrafted stimuli specifically designed to activate targeted regions. However, activity under natural viewing conditions is known to be different [21]. Recent efforts using artificial neural networks as image-computable encoders/predictors of the visual pathway [22, 23, 24, 25, 26, 27, 28, 29, 30] have facilitated the use of more naturalistic stimulus sets. Our proposed method incorporates an image-computable encoding model in line with this past work.

Deep Generative Models. The recent rise of learned generative models has enabled sampling from complex high dimensional distributions. Notable approaches include variational autoencoders [31, 32], generative adversarial networks [33], flows [34, 35], and score/energy/diffusion models [36, 37, 38, 39]. It is possible to condition the model on category [40, 41], text [42, 43], or images [44]. Recent diffusion models have been conditioned with brain activations to reconstruct observed images [45, 46, 47, 48, 49]. Unlike BrainDiVE, these approaches tackle reconstruction but not synthesis of novel images that are predicted to activate regions of the brain.

Brain-Conditioned Image Generation. The differentiable nature of deep encoding models inspired work to create images from brain gradients in mice, macaques, and humans [50, 51, 52]. Without constraints, the images recovered are not naturalistic. Other approaches have combined deep generative models with optimization to recover natural images in macaque and humans [10, 11, 9]. Both [11, 9] utilize fMRI brain gradients combined with ImageNet trained BigGAN. In particular [11] performs end-to-end differentiable optimization by assuming a soft relaxation over the 1 , 000 ImageNet classes; while [9] trains an encoder on the NSD dataset [13] and first searches for

top-classes, then performs gradient optimization within the identified classes. Both approaches are restricted to ImageNet images, which are primarily images of single objects. Our work presents major improvements by enabling the use of diffusion models [44] trained on internet-scale datasets [53] over three magnitudes larger than ImageNet. Concurrent work by [54] explore the use of gradients from macaque V4 with diffusion models, however their approach focuses on early visual cortex with grayscale image outputs, while our work focuses on higher-order visual areas and synthesize complex compositional scenes. By avoiding the search-based optimization procedures used in [9], our work is not restricted to images within a fixed class in ImageNet. Further, to the authors' knowledge we are the first work to use image synthesis methods in the identification of functional specialization in sub-parts of ROIs.

gradient

flow

' '

Null text

Denoiser

image

𝒙𝒕

𝒙𝒕-𝟏

Latent

Decoder

Brain

Encoder

Brain region target

RGB Image

…

|

Gradient

| 2

OPA

RSC

PPA

Start

End

3 Methods

We aim to generate stimuli that maximally activate a given region in visual cortex using paired natural image stimuli and fMRI recordings. We first review relevant background information on diffusion models. We then describe how we can parameterize encoding models that map from images to brain data. Finally, we describe how our framework (Figure 2) can leverage brain signals as guidance to diffusion models to synthesize images that activate a target brain region.

3.1 Background on Diffusion Models

Diffusion models enable sampling from a data distribution p ( x ) by iterative denoising. The sampling process starts with x T ∼ N (0 , I ) , and produces progressively denoised samples x T -1 , x T -2 , x T -3 . . . until a sample x 0 from the target distribution is reached. The noise level varies by timestep t , where the sample at each timestep is a weighted combination of x 0 and ϵ ∼ N (0 , I ) , with x t = √ α t x 0 + ϵ √ 1 -α t . The value of α interpolates between N (0 , I ) and p ( x ) .

In the noise prediction setting, an autoencoder network ϵ θ ( x t , t ) is trained using a mean-squared error E ( x,ϵ,t ) [ ∥ ϵ θ ( x t , t ) -ϵ ∥ 2 2 ] . In practice, we utilize a pretrained latent diffusion model (LDM) [44], with learned image encoder E Φ and decoder D Ω , which together act as an autoencoder I ≈ D Ω ( E Φ ( I )) . The diffusion model is trained to sample x 0 from the latent space of E Φ .

3.2 Brain-Encoding Model Construction

A learned voxel-wise brain encoding model is a function M θ that maps an image I ∈ R 3 × H × W to the corresponding brain activation fMRI beta values represented as an N element vector B ∈ R N : M θ ( I ) ⇒ B . Past work has identified later layers in neural networks as the best predictors of higher visual cortex [30, 56], with CLIP trained networks among the highest performing brain

encoders [28, 57]. As our target is the higher visual cortex, we utilize a two component design for our encoder. The first component consists of a CLIP trained image encoder which outputs a K dimensional vector as the latent embedding. The second component is a linear adaptation layer W ∈ R N × K , b ∈ R N , which maps euclidean normalized image embeddings to brain activation.

Optimal W ∗ , b ∗ are found by optimizing the mean squared error loss over images. We observe that use of a normalized CLIP embedding improves stability of gradient magnitudes w.r.t. the image.

3.3 Brain-Guided Diffusion Model

BrainDiVE seeks to generate images conditioned on maximizing brain activation in a given region. In conventional text-conditioned diffusion models, the conditioning is done in one of two ways. The first approach modifies the function ϵ θ to further accept a conditioning vector c , resulting in ϵ θ ( x t , t, c ) . The second approach uses a contrastive trained image-to-concept encoder, and seeks to maximize a similarity measure with a text-to-concept encoder.

Conditioning on activation of a brain region using the first approach presents difficulties. We do not know a priori the distribution of other non-targeted regions in the brain when a target region is maximized. Overcoming this problem requires us to either have a prior p ( B ) that captures the joint distribution for all voxels in the brain, to ignore the joint distribution that can result in catastrophic effects, or to use a handcrafted prior that may be incorrect [47]. Instead, we propose to condition the diffusion model via our image-to-brain encoder. During inference we perturb the denoising process using the gradient of the brain encoder maximization objective, where γ is a scale, and S ⊆ N are the set of voxels used for guidance. We seek to maximize the average activation of S predicted by M θ :

Like [14, 58, 59], we observe that convergence using the current denoised x t is poor without changes to the guidance. This is because the current image (latent) is high noise and may lie outside of the natural image distribution. We instead use a weighted reformulation with an euler approximation [55, 59] of the final image:

By combining an image diffusion model with a differentiable encoding model of the brain, we are able to generate images that seek to maximize activation for any given brain region.

4 Results

In this section, we use BrainDiVE to highlight the semantic selectivity of pre-identified categoryselective voxels. We then show that our model can capture subtle differences in response properties between ROIs belonging to the same broad category-selective network. Finally, we utilize BrainDiVE to target finer-grained sub-regions within existing ROIs, and show consistent divisions based on semantic and visual properties. We quantify these differences in selectivity across regions using human perceptual studies, which confirm that BrainDiVE images can highlight differences in tuning properties. These results demonstrate how BrainDiVE can elucidate the functional properties of human cortical populations, making it a promising tool for exploratory neuroscience.

4.1 Setup

We utilize the Natural Scenes Dataset (NSD; [13]), which consists of whole-brain 7T fMRI data from 8 human subjects, 4 of whom viewed 10 , 000 natural scene images repeated 3 × . These subjects, S1, S2, S5, and S7, are used for analyses in the main paper (see Supplemental for results for additional subjects). All images are from the MS COCO dataset. We use beta-weights (activations) computed using GLMSingle [60] and further normalize each voxel to µ = 0 , σ = 1 on a per-session basis. We average the fMRI activation across repeats of the same image within a subject. The ∼ 9 , 000 unique images for each subject ([13]) are used to train the brain encoder for each subject, with the remaining ∼ 1 , 000 shared images used to evaluate R 2 . Image generation is on a per-subject basis and done

on an Nvidia V100 using 1 , 500 compute hours. As the original category ROIs in NSD are very generous, we utilize a stricter t > 2 threshold to reduce overlap unless otherwise noted. The final category and ROI masks used in our experiments are derived from the logical AND of the official NSD masks with the masks derived from the official t -statistics.

We utilize stable-diffusion-2-1-base , which produces images of 512 × 512 resolution using ϵ -prediction. Following best practices, we use multi-step 2nd order DPM-Solver++ [61] with 50 steps and apply 0 . 75 SAG [62]. We set step size hyperparameter γ = 130 . 0 . Images are resized to 224 × 224 for the brain encoder. '' (null prompt) is used as the input prompt, thus the diffusion performs unconditional generation without brain guidance. For the brain encoder we use ViT-B/16 , for CLIP probes we use CoCa ViT-L/14 . These are the highest performing LAION-2B models of a given size provided by OpenCLIP [63, 64, 65, 66]. We train our brain encoders on each human subject separately to predict the activation of all higher visual cortex voxels. See Supplemental for visualization of test time brain encoder R 2 . To compare images from different ROIs and sub-regions (OFA/FFA in 4.3, two clusters in 4.4), we asked human evaluators select which of two image groups scored higher on various attributes. We used 100 images from each group randomly split into 10 nonoverlapping subgroups. Each human evaluator performed 80 comparisons, across 10 splits, 4 NSD subjects, and for both fMRI and generated images. See Supplemental for standard error of responses. Human evaluators provided written informed consent and were compensated at $12 . 00 /hour. The study protocol was approved by the institutional review board at the authors' institution.

4.2 Broad Category-Selective Networks

In this experiment, we target large groups of category-selective voxels which can encompass more than one ROI (Figure 3). These regions have been previously identified as selective for broad semantic categories, and this experiment validates our method using these identified regions. The face-, place-, body-, and word- selective ROIs are identified with standard localizer stimuli [67]. The food-selective voxels were obtained from [5].

ventral

expanded

bodies

faces

places

words

food

multiple

The same voxels were used to select the top activating NSD images (referred to as 'NSD') and to guide the generation of BrainDiVE images.

In Figures 4 we visualize, for place-, face-, word-, and body- selective voxels, the top5 out of 10 , 000 images from the fMRI stimulus set (NSD), and the top5 images out of 1 , 000 total images as evaluated by the encoding component of BrainDiVE. For food selective voxels, the top10 are visualized. A visual inspection indicates that our method is able to generate diverse images that semantically represent the target category. We further use CLIP to perform semantic probing of the images, and force the images to be classified into one of five categories. We measure the percentage of images that match the preferred category for a given set of voxels (Table 1). We find that our top10% and 20% of images exceed the top1% and 2% of natural images in accuracy, indicating our method has high semantic specificity.

Place voxels

NSD

(fMRI)

BrainDiVE

(Ours)

Word voxels

Face voxels

NSD

(fMRI)

BrainDiVE

(Ours)

Body voxels

Food voxels

NSD

(fMRI)

BrainDiVE

(Ours)

aeetahut

PIRASF

NO100

CETFCD

DOD

ODRE

HO

1ON

2116

4.3 Individual ROIs

In this section, we apply our method to individual ROIs that are selective for the same broad semantic category. We focus on the occipital face area (OFA) and fusiform face area (FFA), as initial tests suggested little differentiation between ROIs within the place-, word-, and body- selective networks. In this experiment, we also compare our results against the top images for FFA and OFA from NeuroGen [9], using the top 100 out of 500 images provided by the authors. Following NeuroGen, we also generate 500 total images, targeting FFA and OFA separately (Figure 5). We observe that both diffusion-generated and NSD images have very high face content in FFA, whereas NeuroGen has higher animal face content. In OFA, we observe both NSD and BrainDiVE images have a strong face component, although we also observe text selectivity in S2 and animal face selectivity in S5. Again NeuroGen predicts a higher animal component than face for S5. By avoiding the use of fixed categories, BrainDiVE images are more diverse than those of NeuroGen. This trend of face and animals appears at t > 2 and the much stricter t > 5 threshold for identifying faceselective voxels ( t > 5 used for visualization/evaluation). The differences in images synthesized by BrainDiVE for FFA and OFA are consistent with past work suggesting that FFA represents faces at a higher level of abstraction than OFA, while OFA shows greater selectivity to low-level face features and sub-components, which could explain its activation by off-target categories [68, 69, 70].

To quantify these results, we perform a human study where subjects are asked to compare the top-100 images between FFA & OFA, for both NSD and generated images. Results are shown in Table 2.

S2 FFA

S2 OFA

NSD

(fMRI)

BrainDiVE

(Ours 1-5)

BrainDiVE

(Ours 6-10)

NeuroGen

(BigGAN)

NSD

(fMRI)

NeuroGen

(BigGAN)

BrainDiVE

(Ours 6-10)

BrainDiVE

(Ours 1-5)

S5 FFA

S5 OFA

STOP

Peuoces

46

ALLBAY

OrDue

QO

We find that OFA consistently has higher animal and abstract content than FFA. Most notably, this difference is on average more pronounced in the images from BrainDiVE, indicating that our approach is able to highlight subtle differences in semantic selectivity across regions.

S5 food

S1 food

S5 OPA

S1 OPA

food 1

food 2

OPA 1

OPA 2

4.4 Semantic Divisions within ROIs

In this experiment, we investigate if our model can identify novel sub-divisions within existing ROIs. We first perform clustering on normalized per-voxel encoder weights using vmf-clustering [71]. We find consistent cosine difference between the cluster centers in the food-selective ROI as well as in the occipital place area (OPA), clusters shown in Figure 6. In all four subjects, we observe a relatively consistent anterior-posterior split of OPA. While the clusters within the food ROI vary more anatomically, each subject appears to have a more medial and a more lateral cluster. We visualize the images for the two food clusters in Figure 7, and for the two OPA clusters in Figure 8. We observe that for both the food ROI and OPA, the BrainDiVE-generated images from each cluster have noticeable differences in their visual and semantic properties. In particular, the BrainDiVE images from food cluster-2 have much higher color saturation than those from cluster-1, and also have more objects

Food cluster 1 NSD (fMRI)

S1

S2

S1

S2

Food cluster 2 NSD (fMRI)

Food cluster 1 BrainDiVE

Food cluster 2 BrainDiVE

that resemble fruits and vegetables. In contrast, food cluster-1 generally lacks vegetables and mostly consist of bread-like foods. In OPA, cluster-1 is dominated by indoor scenes (rooms, hallways), while 2 is overwhelmingly outdoor scenes, with a mixture of natural and man-made structures viewed from a far perspective. Some of these differences are also present in the NSD images, but the differences appear to be highlighted in the generated images.

To confirm these effects, we perform a human study (Table 3, Table 4) comparing the images from different clusters in each ROI, for both NSD and generated images. As expected from visual inspection of the images, we find that food cluster-2 is evaluated to have higher vegetable/fruit content, judged to be healthier, more colorful, and slightly more distant than food cluster-1. We find that OPA cluster-1 is evaluated to be more angular/geometric, include more indoor scenes, to be less natural and consisting of less distant scenes. Again, while these trends are present in the NSD images, they are more pronounced with the BrainDiVE images. This not only suggests that our method has uncovered differences in semantic selectivity within pre-existing ROIs, but also reinforces the ability of BrainDiVE to identify and highlight core functional differences across visual cortex regions.

OPA cluster 2 NSD (fMRI)

OPA cluster 1 BrainDiVE

OPA cluster 2 BrainDiVE

S7

S2

S7

S2

OPA cluster 1 NSD (fMRI)

5 Discussion

Limitations and Future Work Here, we show that BrainDiVE generates diverse and realistic images that can probe the human visual pathway. This approach relies on existing large datasets of natural images paired with brain recordings. In that the evaluation of synthesized images is necessarily qualitative, it will be important to validate whether our generated images and candidate features derived from these images indeed maximize responses in their respective brain areas. As such, future work should involve the collection of human fMRI recordings using both our synthesized images and more focused stimuli designed to test our qualitative observations. Future work may also explore the images generated when BrainDiVE is applied to additional sub-region, new ROIs, or mixtures of ROIs.

Conclusion We introduce a novel method for guiding diffusion models using brain activations BrainDiVE - enabling us to leverage generative models trained on internet-scale image datasets for

data driven explorations of the brain. This allows us to better characterize fine-grained preferences across the visual system. We demonstrate that BrainDiVE can accurately capture the semantic selectivity of existing characterized regions. We further show that BrainDiVE can capture subtle differences between ROIs within the face selective network. Finally, we identify and highlight finegrained subdivisions within existing food and place ROIs, differing in their selectivity for mid-level image features and semantic scene content. We validate our conclusions with extensive human evaluation of the images.

References

Kalanit Grill-Spector and Rafael Malach. The human visual cortex. Annual Review of Neuroscience , 27:649-677, 2004. ISSN 0147006X. doi: 10.1146/ANNUREV.NEURO.27.070203. 144220.

J Sergent, S Ohta, and B MacDonald. Functional neuroanatomy of face and object processing: A positron emission tomography study. Brain , 115:15-36, 1992.

Nancy Kanwisher, Josh McDermott, and Marvin M Chun. The fusiform face area: a module in human extrastriate cortex specialized for face perception. Journal of neuroscience , 17(11): 4302-4311, 1997.

Russell Epstein and Nancy Kanwisher. A cortical representation of the local visual environment. Nature , 392(6676):598-601, 1998.

Nidhi Jain, Aria Wang, Margaret M. Henderson, Ruogu Lin, Jacob S. Prince, Michael J. Tarr, and Leila Wehbe. Selectivity for food in human ventral visual cortex. Communications Biology 2023 6:1 , 6:1-14, 2 2023. ISSN 2399-3642. doi: 10.1038/s42003-023-04546-2.

Ian M L Pennock, Chris Racey, Emily J Allen, Yihan Wu, Thomas Naselaris, Kendrick N Kay, Anna Franklin, and Jenny M Bosten. Color-biased regions in the ventral visual pathway are food selective. Curr. Biol. , 33(1):134-146.e4, 2023.

Meenakshi Khosla, N. Apurva Ratan Murty, and Nancy Kanwisher. A highly selective response to food in human visual cortex revealed by hypothesis-free voxel decomposition. Current Biology , 32:1-13, 2022.

A Ishai, L G Ungerleider, A Martin, J L Schouten, and J V Haxby. Distributed representation of objects in the human ventral visual pathway. Proc Natl Acad Sci U S A , 96(16):9379-9384, 1999.

Zijin Gu, Keith Wakefield Jamison, Meenakshi Khosla, Emily J Allen, Yihan Wu, Ghislain St-Yves, Thomas Naselaris, Kendrick Kay, Mert R Sabuncu, and Amy Kuceyeski. NeuroGen: activation optimized image synthesis for discovery neuroscience. NeuroImage , 247:118812, 2022.

Carlos R Ponce, Will Xiao, Peter F Schade, Till S Hartmann, Gabriel Kreiman, and Margaret S Livingstone. Evolving images for visual neurons using a deep generative network reveals coding principles and neuronal preferences. Cell , 177(4):999-1009, 2019.

N Apurva Ratan Murty, Pouya Bashivan, Alex Abate, James J DiCarlo, and Nancy Kanwisher. Computational models of category-selective brain regions enable high-throughput tests of selectivity. Nature communications , 12(1):5540, 2021.

Nadine Chang, John A Pyles, Austin Marcus, Abhinav Gupta, Michael J Tarr, and Elissa M Aminoff. Bold5000, a public fMRI dataset while viewing 5000 visual images. Scientific Data , 6(1):1-18, 2019.

Emily J Allen, Ghislain St-Yves, Yihan Wu, Jesse L Breedlove, Jacob S Prince, Logan T Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, et al. A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence. Nature neuroscience , 25(1):116-126, 2022.

Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint arXiv:2112.10741 , 2021.

Robert Desimone, Thomas D Albright, Charles G Gross, and Charles Bruce. Stimulus-selective properties of inferior temporal neurons in the macaque. Journal of Neuroscience , 4(8):20512062, 1984.

Laurent Cohen, Stanislas Dehaene, Lionel Naccache, Stéphane Lehéricy, Ghislaine DehaeneLambertz, Marie-Anne Hénaff, and François Michel. The visual word form area: spatial and temporal characterization of an initial stage of reading in normal subjects and posterior split-brain patients. Brain , 123(2):291-307, 2000.

Paul E Downing, Yuhong Jiang, Miles Shuman, and Nancy Kanwisher. A cortical area selective for visual processing of the human body. Science , 293(5539):2470-2473, 2001.

Bruce D McCandliss, Laurent Cohen, and Stanislas Dehaene. The visual word form area: expertise for reading in the fusiform gyrus. Trends in cognitive sciences , 7(7):293-299, 2003.

Meenakshi Khosla, N Apurva Ratan Murty, and Nancy Kanwisher. A highly selective response to food in human visual cortex revealed by hypothesis-free voxel decomposition. Current Biology , 32(19):4159-4171, 2022.

Ian ML Pennock, Chris Racey, Emily J Allen, Yihan Wu, Thomas Naselaris, Kendrick N Kay, Anna Franklin, and Jenny M Bosten. Color-biased regions in the ventral visual pathway are food selective. Current Biology , 33(1):134-146, 2023.

Jack L Gallant, Charles E Connor, and David C Van Essen. Neural activity in areas v1, v2 and v4 during free viewing of natural scenes compared to controlled viewing. Neuroreport , 9(7): 1673-1678, 1998.

Thomas Naselaris, Kendrick N Kay, Shinji Nishimoto, and Jack L Gallant. Encoding and decoding in fmri. Neuroimage , 56(2):400-410, 2011.

Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James J DiCarlo. Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences , 111(23):8619-8624, 2014.

Seyed-Mahdi Khaligh-Razavi and Nikolaus Kriegeskorte. Deep supervised, but not unsupervised, models may explain it cortical representation. PLoS computational biology , 10(11): e1003915, 2014.

Michael Eickenberg, Alexandre Gramfort, Gaël Varoquaux, and Bertrand Thirion. Seeing it all: Convolutional network layers map the function of the human visual system. NeuroImage , 152: 184-194, 2017.

Haiguang Wen, Junxing Shi, Wei Chen, and Zhongming Liu. Deep residual network predicts cortical representation and organization of visual features for rapid categorization. Scientific reports , 8(1):3752, 2018.

Jonas Kubilius, Martin Schrimpf, Kohitij Kar, Rishi Rajalingham, Ha Hong, Najib Majaj, Elias Issa, Pouya Bashivan, Jonathan Prescott-Roy, Kailyn Schmidt, et al. Brain-like object recognition with high-performing shallow recurrent anns. Advances in neural information processing systems , 32, 2019.

Colin Conwell, Jacob S Prince, George Alvarez, and Talia Konkle. Large-scale benchmarking of diverse artificial vision models in prediction of 7t human neuroimaging data. bioRxiv , pages 2022-03, 2022.

Tom Dupré la Tour, Michael Eickenberg, Anwar O Nunez-Elizalde, and Jack L Gallant. Featurespace selection with banded ridge regression. NeuroImage , 264:119728, 2022.

Aria Yuan Wang, Kendrick Kay, Thomas Naselaris, Michael J Tarr, and Leila Wehbe. Incorporating natural language into vision models improves prediction and understanding of higher visual cortex. BioRxiv , pages 2022-09, 2022.

Diederik P Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv:1312.6114 , 2013.

Aaron Van Den Oord, Oriol Vinyals, et al. Neural discrete representation learning. Advances in neural information processing systems , 30, 2017.

Ian Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron Courville, and Yoshua Bengio. Generative adversarial networks. Communications of the ACM , 63(11):139-144, 2020.

Danilo Rezende and Shakir Mohamed. Variational inference with normalizing flows. In International conference on machine learning , pages 1530-1538. PMLR, 2015.

Laurent Dinh, David Krueger, and Yoshua Bengio. Nice: Non-linear independent components estimation. arXiv preprint arXiv:1410.8516 , 2014.

Aapo Hyvärinen and Peter Dayan. Estimation of non-normalized statistical models by score matching. Journal of Machine Learning Research , 6(4), 2005.

Jascha Sohl-Dickstein, Eric Weiss, Niru Maheswaranathan, and Surya Ganguli. Deep unsupervised learning using nonequilibrium thermodynamics. In International Conference on Machine Learning , pages 2256-2265. PMLR, 2015.

Yang Song, Jascha Sohl-Dickstein, Diederik P Kingma, Abhishek Kumar, Stefano Ermon, and Ben Poole. Score-based generative modeling through stochastic differential equations. arXiv preprint arXiv:2011.13456 , 2020.

Jonathan Ho, Chitwan Saharia, William Chan, David J Fleet, Mohammad Norouzi, and Tim Salimans. Cascaded diffusion models for high fidelity image generation. J. Mach. Learn. Res. , 23(47):1-33, 2022.

Mehdi Mirza and Simon Osindero. Conditional generative adversarial nets. arXiv preprint arXiv:1411.1784 , 2014.

Andrew Brock, Jeff Donahue, and Karen Simonyan. Large scale gan training for high fidelity natural image synthesis. arXiv preprint arXiv:1809.11096 , 2018.

Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text-conditional image generation with clip latents. arXiv preprint arXiv:2204.06125 , 2022.

Scott Reed, Zeynep Akata, Xinchen Yan, Lajanugen Logeswaran, Bernt Schiele, and Honglak Lee. Generative adversarial text to image synthesis. In International conference on machine learning , pages 1060-1069. PMLR, 2016.

Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. Highresolution image synthesis with latent diffusion models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 10684-10695, 2022.

Zijiao Chen, Jiaxin Qing, Tiange Xiang, Wan Lin Yue, and Juan Helen Zhou. Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding. arXiv preprint arXiv:2211.06956 , 1(2):4, 2022.

Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. bioRxiv , pages 2022-11, 2022.

Furkan Ozcelik and Rufin VanRullen. Brain-diffuser: Natural scene reconstruction from fmri signals using generative latent diffusion. arXiv preprint arXiv:2303.05334 , 2023.

Yizhuo Lu, Changde Du, Dianpeng Wang, and Huiguang He. Minddiffuser: Controlled image reconstruction from human brain activity with semantic and structural diffusion. arXiv preprint arXiv:2303.14139 , 2023.

Reese Kneeland, Jordyn Ojeda, Ghislain St-Yves, and Thomas Naselaris. Second sight: Using brain-optimized encoding models to align image distributions with human brain activity. ArXiv , 2023.

Edgar Y Walker, Fabian H Sinz, Erick Cobos, Taliah Muhammad, Emmanouil Froudarakis, Paul G Fahey, Alexander S Ecker, Jacob Reimer, Xaq Pitkow, and Andreas S Tolias. Inception loops discover what excites neurons most using deep predictive models. Nature neuroscience , 22(12):2060-2065, 2019.

Pouya Bashivan, Kohitij Kar, and James J DiCarlo. Neural population control via deep image synthesis. Science , 364(6439):eaav9436, 2019.

Meenakshi Khosla and Leila Wehbe. High-level visual areas act like domain-general filters with strong selectivity and functional specialization. bioRxiv , pages 2022-03, 2022.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion5b: An open large-scale dataset for training next generation image-text models. arXiv preprint arXiv:2210.08402 , 2022.

Paweł A Pierzchlewicz, Konstantin F Willeke, Arne F Nix, Pavithra Elumalai, Kelli Restivo, Tori Shinn, Cate Nealley, Gabrielle Rodriguez, Saumil Patel, Katrin Franke, et al. Energy guided diffusion for generating neurally exciting images. bioRxiv , 2023.

Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502 , 2020.

Aria Y. Wang, Ruogu Lin, Michael J. Tarr, and Leila Wehbe. Joint interpretation of representations in neural network and the brain. In 'How Can Findings About The Brain Improve AI Systems?' Workshop @ ICLR 2021 , 2021.

Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang, and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint arXiv:2303.15389 , 2023.

Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Advances in Neural Information Processing Systems , 34:8780-8794, 2021.

Wei Li, Xue Xu, Xinyan Xiao, Jiachen Liu, Hu Yang, Guohao Li, Zhanpeng Wang, Zhifan Feng, Qiaoqiao She, Yajuan Lyu, et al. Upainting: Unified text-to-image diffusion generation with cross-modal guidance. arXiv preprint arXiv:2210.16031 , 2022.

Jacob S Prince, Ian Charest, Jan W Kurzawski, John A Pyles, Michael J Tarr, and Kendrick N Kay. Improving the accuracy of single-trial fmri response estimates using glmsingle. eLife , 11: e77599, nov 2022. ISSN 2050-084X. doi: 10.7554/eLife.77599.

Cheng Lu, Yuhao Zhou, Fan Bao, Jianfei Chen, Chongxuan Li, and Jun Zhu. Dpmsolver++: Fast solver for guided sampling of diffusion probabilistic models. arXiv preprint arXiv:2211.01095 , 2022.

Susung Hong, Gyuseong Lee, Wooseok Jang, and Seungryong Kim. Improving sample quality of diffusion models using self-attention guidance. arXiv preprint arXiv:2210.00939 , 2022.

Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML , 2021.

Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu. Coca: Contrastive captioners are image-text foundation models. arXiv preprint arXiv:2205.01917 , 2022.

Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade W Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, Patrick Schramowski, Srivatsa R Kundurthy, Katherine Crowson, Ludwig Schmidt, Robert Kaczmarczyk, and Jenia Jitsev. LAION-5b: An open large-scale dataset for training next generation image-text models. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track , 2022.

Gabriel Ilharco, Mitchell Wortsman, Ross Wightman, Cade Gordon, Nicholas Carlini, Rohan Taori, Achal Dave, Vaishaal Shankar, Hongseok Namkoong, John Miller, Hannaneh Hajishirzi, Ali Farhadi, and Ludwig Schmidt. Openclip, July 2021.

A. Stigliani, K. S. Weiner, and K. Grill-Spector. Temporal processing capacity in high-level visual cortex is domain specific. Journal of Neuroscience , 35:12412-12424, 2015. ISSN 0270-6474. doi: 10.1523/JNEUROSCI.4822-14.2015.

Jia Liu, Alison Harris, and Nancy Kanwisher. Perception of face parts and face configurations: An fmri study. Journal of cognitive neuroscience , 22:203, 1 2010. ISSN 0898929X. doi: 10.1162/JOCN.2009.21203.

David Pitcher, Vincent Walsh, and Bradley Duchaine. The role of the occipital face area in the cortical face perception network. Experimental brain research , 209:481-493, 4 2011. ISSN 1432-1106. doi: 10.1007/S00221-011-2579-1.

Maria Tsantani, Nikolaus Kriegeskorte, Katherine Storrs, Adrian Lloyd Williams, Carolyn McGettigan, and Lúcia Garrido. Ffa and ofa encode distinct types of face identity information. Journal of Neuroscience , 41:1952-1969, 3 2021. ISSN 0270-6474. doi: 10.1523/JNEUROSCI. 1449-20.2020.

Arindam Banerjee, Inderjit S Dhillon, Joydeep Ghosh, Suvrit Sra, and Greg Ridgeway. Clustering on the unit hypersphere using von mises-fisher distributions. Journal of Machine Learning Research , 6(9), 2005.

Chenlin Meng, Yang Song, Jiaming Song, Jiajun Wu, Jun-Yan Zhu, and Stefano Ermon. Sdedit: Image synthesis and editing with stochastic differential equations. arXiv preprint arXiv:2108.01073 , 2021.

Jun Hao Liew, Hanshu Yan, Daquan Zhou, and Jiashi Feng. MagicMix: Semantic mixing with diffusion models, 2022.

Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 , 2020.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101 , 2017.

James S Gao, Alexander G Huth, Mark D Lescroart, and Jack L Gallant. Pycortex: an interactive surface visualizer for fmri. Frontiers in neuroinformatics , page 23, 2015.

Supplementary Material: Brain Diffusion for Visual Exploration

Broader Impacts (section 7)

Visualization of each subject's category selective voxels (section 8)

CLIP zero-shot classification results for all subjects (section 9)

Image gradients and synthesis process (section 10)

Standard error for human behavioral studies (section 11)

Brain Encoder R 2 (section 12)

Additional OFA and FFA visualizations (section 13)

Additional OPA and food clustering visualizations (section 14)

Training, inference, and experiment details (section 15)

7 Broader impacts

Our work introduces a method where brain responses - as measured by fMRI - can be used to guide diffusion models for image synthesis (BrainDiVE). We applied BrainDiVE to probe the representation of high-level semantic information in the human visual cortex. BrainDiVE relies on pretrained stable-diffusion-2-1 and will necessarily reflect the biases in the data used to train these models. However, given the size and diversity of this training data, BrainDiVE may reveal data-driven principles of cortical organization that are unlikely to have been identified using more constrained, hypothesis-driven experiments. As such, our work advances our current understanding of the human visual cortex and, with larger and more sensitive neuroscience datasets, may be utilized to facilitate future fine-grained discoveries regarding neural coding, which can then be validated using hypothesis-driven experiments.

8 Visualization of each subject's category selective voxel images

Place Voxels

WordVoxels

ITAOS

BrainDiVE

(sno)

aeelahut

PIRASF

NO100

ODRE

DOD

HO

2116

FaceVoxels

BrainDiVE

(suno)

FoodVoxels

BrainDiVE

(sno)

PlaceVoxels

WordVoxels

Bertions

STOP

STOP

CAPMHL

Time

ALUWAY

BrainDiVE

(suno)

Ages

TAkeita

NLIAN

OLAA

010

FATIUS

Face Voxels

BrainDiVE

(sno)

FoodVoxels

BrainDiVE

(suno)

PlaceVoxels

WordVoxels

STOP

NEVER

FREE!

LOKING

BrainDiVE

(sno)

BETARIY

RLEKN

JL

PIEERLSlla

11OANUEEOY

YOAI

2

FaceVoxels

BodyVoxels

BrainDiVE

(sno)

FoodVoxels

NSD

(fMRI)

BrainDiVE

(suno)

PlaceVoxels

WordVoxels

BrainDiVE

SERCEUCT EXHY

(sno)

ISYE

loektfer

mau lbea fmdrilee

W

aa

-dunertt

rounirser

Face Voxels

NSD

(fMRI)

BrainDiVE

Food Voxels

BrainDiVE

(sno)

N

PlaceVoxels

WordVoxels

BrainDiVE

APIRRI

ReuAriub

(suno)

BCrMAS

CAU DTE

PEONTNCE

NEGY

Ceu ewgu Kin

ACRKIN

Face Voxels

BrainDiVE

(suno)

Food Voxels

BrainDiVE

(suno)

PlaceVoxels

WordVoxels

BrainDiVE

(suno)

Tsrlino

元

ROBETSG

Face Voxels

NSD

(fMRI)

BrainDiVE

(sno)

Food Voxels

BrainDiVE

(sno)

PlaceVoxels

WordVoxels

STOP

R

BrainDiVE

MAAN

(sno)

Face Voxels

(fMRI

BrainDiVE

(suno)

FoodVoxels

NSD

(fMRI)

BrainDiVE

(sno)

PlaceVoxels

WordVoxels

TUNDI

STOP

D

PNCBAI

BrainDiVE

(sno)

CAVLL

CALISIA

OINETSIE

Face Voxels

BrainDiVE

(suno)

Food Voxels

BrainDiVE

(suno)

VSR

9 CLIP zero-shot classification

In this section we show the CLIP classification results for S1 - S8, where Table S.1 in this Supplementary material matches that of Table 1 in the main paper. We use CLIP [63] to classify images from each ROI into five semantic categories: face/place/body/word/food. Shown is the percentage where the classified category of the image matches the preferred category of the brain region. We show this for the top-200 and top-100 images (top-2% and top-1%) evaluated by mean true fMRI beta, and the top-200 and top-100 ( 20% and 10% ) of BrainDiVE images as self-evaluated by the encoding component of BrainDiVE. Please see Supplementary Section 15 for the prompts we use for CLIP classification.

10 Image gradients and synthesis process

In this section, we show examples of the image at each step of the synthesis process. We perform this visualization for face-, place-, body-, word-, and food- selective voxels. Two visualizations are shown for each set of voxels, we use S1 for all visualizations in this section. The diffusion model is guided only by the objective of maximizing a given set of voxels. We observe that coarse image structure emerges very early on from brain guidance. Furthermore, the gradient and diffusion model sometimes work against each other. For example in Figure S.14 for body voxels, the brain gradient induces the addition of an extra arm, while the diffusion has already generated three natural bodies. Or in Figure S.15 for word voxels, where the brain gradient attempts to add horizontal words, but they are warped by the diffusion model. Future work could explore early guidance only, as described in 'SDEdit' and 'MagicMix' [72, 73].

10.1 Face voxels

We show examples where the end result contains multiple faces (Figure S.9), or a single face (Figure S.10).

95333535338339

t=1...10

t=11...20

t=21...30

t=31...40

t=1...10

t=11...20

t=21...30

t=31...40

10.2 Place voxels

We show examples where the end result contains an indoor scene (Figure S.11), or an outdoor scene (Figure S.12).

t=1...10

t=11...20

t=21...30

t=31...40

t=1...10

t=11...20

t=21...30

t=31...40

10.3 Body voxels

We show examples where the end result contains an single person's body (Figure S.13), or an multiple people (Figure S.14).

t=1...10

t=11...20

t=21...30

t=31...40

t=1...10

t=11...20

t=21...30

t=31...40

10.4 Word voxels

We show examples where the end result contains recognizable words (Figure S.15), or glyph like objects (Figure S.16).

1.

t=1...10

t=11...20

TRLEBAFYS

HQURNAEYS

HQLEbAFYS

HQLPBAFYS

HQLPbAEYS

RIIES

RUPIES

RPIES

RUFIES

TUf16S

RUFIES

t=21...30

HQLPbAEYS

HQLPbAEYS

HQLPbAEYS

HQLPbAEYS

HQLPbAEYS

HQLPbAEYS

HQLPbAEYS

HQLPbAEYS

HQLPbAEYS

RUFIES

RUFIES

RUFIES

RUF1ES

RUFIES

RUFIES

RUFIES

RUFIES

RFIES

RUFIES

t=31...40

HQLPbAEYS

HQLPbAEYS

HQLPbAEYS

HQLPbAEYS

HQLPbAEYS

HQLPbAEYS

HQLPbAEYS

HQLPbAEYS

HQLPbAEYS

HQLPbAEYS

RF16S

RF16S

RUF:16S

RUF:16S

RUT16S

HQLPbAEYS

HQLPbAEYS

HQLPbAEYS

HQLPbAEYS

HQLPbAEYS

HQLPbAEYS

HQLPbAEYS

HQLPbAEYS

HQLPbAEYS

HQLPbAEYS

RUF:16S

RUT:16S

RUF:16S

RUF:16S

RUF:16S

RUT:16S

RUT:16S

RUT16S

t=1...10

t=11...20

t=21...30

o1o

t=31...40

10.5 Food voxels

We show examples where the end result contains highly processed foods (Figure S.17, showing what appears to be a cake), or cooked food containing vegetables (Figure S.18).

t=1...10

t=11...20

t=21...30

t=31...40

t=1...10

t=11...20

t=21...30

t=31...40

11 Human behavioral study standard error

In this section, we show the human behavioral study results along with the standard error of the responses. Each question was answered by exactly 10 subjects from prolific.co . In each table, the results are show in the following format: Mean(SEM) . Where Mean is the average response, while SEM is the standard error of the mean ratio across 10 subjects: (SEM = σ √ 10 ).

Which cluster is more...

OPA-1 NSD

OPA-2 NSD

OPA-1 BrainDiVE

OPA-2 BrainDiVE

S1

14(3.8)

73

(3.4)

6(3.2)

91

(5.7)

natural

S2

3(2.0)

89

(7.4)

6(1.5)

91

(3.6)

S5

9(4.1)

71

(6.4)

9(3.6)

83

(6.9)

S7

10(2.8)

(6.1)

81

6(2,9)

(5.5)

90

S1

10(2.4)

69

(4.6)

1(0.9)

97

(2.8)

far away

S2

S5

1(0.9)

93

(3.8)

3(2.8)

92

(6.6)

6(2.9)

81

(6.5)

3(2.8)

91

(5.6)

S7

8(2.4)

85

(5.5)

8(5.6)

88

(7.4)

12 Brain encoder R 2

S1

S2

S3

S5

S6

S7

S8

In Figure 12 we show the R 2 of the brain encoder as evaluated on the test images. Our brain encoder consists of a CLIP backbone and a linear adaptation layer. We do not model voxels in the early visual cortex, nor do we model voxels outside of higher visual. Our model can generally achieve high R 2 in regions in known regions of visual semantic selectivity.

13 OFA and FFA visualizations

In this section, we visualize the top10 NSD and BrainDiVE images for OFA and FFA. NSD images are selected using the fMRI betas averaged within each ROI. BrainDiVE images are ranked using our predicted ROI activities from 500 images.

S2 FFA

(fMRI)

BrainDivE

(suno)

D

(fMRI

BrainDiVE

(suno)

(fMRI)

BrainDiVE

(no

S2 OFA

STOP

STOP

S

D

(MRI

46

Time

BrainDiVE

(suno)

S5 FFA

S7 FFA

D

R

S

BrainDiVE

(suno)

JF

S5 OFA

D

R

S

BrainDiVE

(Ours)

口

S

GREEN

ONST

BrainDiVE

(suno)

S7 OFA

D

(fMRI

BrainDiVE

(uno)

14 OPA and food visualizations

S5 food

S1 food

food 1

food 2

OPA 1

OPA 2

S2 food

S7 food

S5 OPA

S2 OPA

S1 OPA

S7 OPA

心

Consistent with Jain et al. [5], we observe that the food voxels themselves are anatomically variable across subjects, while the two food clusters form alternating patches within the food patches. OPA generally yields anatomically consistent clusters in the four subjects we investigated, with all four subjects showing an anterior-posterior split for OPA.

OPA cluster 1 NSD (fMRI)

OPA cluster 2 NSD (fMRI)

OPA cluster 2 BrainDiVE

S2

S2

S1

S1

OPA cluster 1 BrainDiVE

OPA cluster 2 NSD (fMRI)

OPA cluster 1 BrainDiVE

OPA cluster 2 BrainDiVE

S5

S7

S7

OPA cluster 1 NSD (fMRI)

S5

Food cluster 1 NSD (fMRI)

S1

S2

S1

S2

Food cluster 2 NSD (fMRI)

Food cluster 1 BrainDiVE

Food cluster 2 BrainDiVE

S7

S5

S7

Food cluster 1 NSD (fMRI)

Food cluster 1 BrainDiVE

S5

Food cluster 2 NSD (fMRI)

Food cluster 2 BrainDiVE

9101

15 Training, inference, and compute details

Encoder training. Our encoder backbone uses ViT-B/16 with CLIP pretrained weights laion2b_s34b_b88k provided by OpenCLIP [65, 66]. The ViT [74] weights for the brain encoder are frozen. We train a linear layer consisting of weight and bias to map from the 512 dimensional vector to higher visual voxels B . The CLIP image branch outputs are normalized to the unit sphere.

Training is done using the Adam optimizer [75] with learning rate lr init = 3 e -4 and lr end = 1 . 5 e -4 , with learning rate adjusting exponentially each epoch. We train for 100 epochs. Decoupled weight decay [76] of magnitude decay = 2 e -2 is applied. Each subject is trained independently using the ∼ 9000 images unique to each subject's stimulus set, with R 2 evaluated on the ∼ 1000 images shared by all subjects.

During training of the encoder weights, the image is resized to 224 × 224 to match the input size of ViT-B/16 . We augment the images by first randomly scaling the pixels by a value between [0 . 95 , 1 . 05] , then normalize the image using OpenCLIP ViT image mean and variance. Prior to input to the network, we further randomly offset the image spatially by up to 4 pixels along the height and width dimensions. The empty pixels are filled in using edge value padding. A small amount of gaussian noise N (0 , 0 . 05 2 ) is added to each pixel prior to input to the encoder backbone.

Objective. For all experiments, the objective used is the maximization of a selected set of voxels. Here we will further draw a link between the optimization objective we use and the traditional CLIP text prompt guidance objective [14, 59]. Recall that M θ is our brain activation encoder that maps from the image to per-voxel activations. It accepts as input an image, passes it through a ViT backbone, normalizes that vector to the unit sphere, then applies a linear mapping to go to per-voxel activations. S ∈ N are the set of voxels we are currently trying to maximize (where N is the set of all voxels in the brain), γ is a step size parameter, and D Ω is the decoder from the latent diffusion model that outputs an RGB image (we ignore the euler approximation for clarity). Also recall that we use a diffusion model that performs ϵ -prediction.

In the general case, we perturb the denoising process by trying to maximize a set of voxels S :

For the purpose of this section, we will focus on a single voxel first, then discuss the multi-voxel objective.

In our case, the single voxel perturbation is (assuming W is a vector, and that ⟨· , ·⟩ is the inner product):

We can ignore b, as it does not affect optimal CLIP img ( I gen )

Now let us consider the typical CLIP guidance objective for diffusion models, where P text is the guidance prompt, and CLIP text is the text encoder component of CLIP:

̸

As such, the W that we find by linearly fitting CLIP image embeddings to brain activation plays the role of a text prompt. In reality, ∥ W ∥ 2 = 1 (but norm is a constant for each voxel), and there is likely

no computationally efficient way to 'invert' W directly into a human interpretable text prompt. By performing brain guidance, we are essentially using the diffusion model to synthesize an image I gen where in addition to satisfying the natural image constraint, the image also attempts to satisfy:

Or put another way, it generates images where the CLIP latent is aligned with the direction of W . Let us now consider the multi-voxel perturbation, where W i , b i is the per-voxel weight vector and bias:

We move | S | outside of the gradient operation

We again ignore b i as it does not affect gradient

We can move ∑ outside due to the distributive nature of gradients

Thus from a gradient perspective, the total gradient is the average of gradients from all voxels. Recall that the inner product is a bilinear function, and that the CLIP image latent is on the unit sphere. Then we are generating an image that

Where the optimal image has a CLIP latent that is aligned with the direction of ∑ i ∈ S W i .

Compute. We perform our experiments on a cluster of Nvidia V100 GPUs in either 16GB or 32GB VRAM configuration, and all experiments consumed approximately 1 , 500 compute hours. Each image takes between 20 and 30 seconds to synthesize. All experiments were performed using PyTorch, with cortex visualizations done using PyCortex [77].

CLIP prompts. Here we list the text prompts that are used to classify the images for Table 1. in the main paper.

face_class = ["A face facing the camera", "A photo of a face", "A photo of a human face", "A photo of faces", "A photo of a person's face", "A person looking at the camera", "People looking at the camera","A portrait of a person", "A portrait photo"]

body_class = ["A photo of a torso", "A photo of torsos", "A photo of limbs", "A photo of bodies", "A photo of a person", "A photo of people"]

scene_class = ["A photo of a bedroom", "A photo of an office","A photo of a hallway", "A photo of a doorway", "A photo of interior design", "A photo

of a building", "A photo of a house", "A photo of nature", "A photo of landscape", "A landscape photo", "A photo of trees", "A photo of text_class = ["A photo of words", "A photo of glyphs", "A photo of a "A photo of text", "A photo of numbers", "A photo of a letter", "A photo of letters", "A photo of writing", "A photo of text on an object"]

We classify an image as belonging to a category if the image's CLIP latent has highest cosine similarity with the CLIP latent of a prompt belonging to a given category. The same prompts are used to classify the NSD and generated images.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a diffusion-based method for fMRI-guided image synthesis and claims that it can identify the selectivity of various ROIs in the visual cortex. The motivation of this work is well-defined and is of vital importance to the computational neuroscience field. While the method in this work is intuitive and lacks machine learning depth.\n\nSTRENGTHS:\nStrengths: \n* Interesting scientific field and question of neuroscience.\n* The paper is well-written and has a smooth and concrete flow.\n* The experiments and visualizations are in good condition and sufficient.\n\nWEAKNESSES:\nWeaknesses\n\n* The method used in this paper is heuristic and lacks technical depth. \n    * Generating or reconstructing images from fMRI is not a new idea, as has been proposed in [1], etc.\n    * CLIP is an off-the-shelf large-scale pre-trained model, performing conditional diffusion-based generation with embedding from CLIP or Stable Diffusion is stereotyped [2,3].\n    * In Section 3.3, the domain-specific techniques you designed for this specific task are just some first-order tricks (i.e., linear combination of S, euler approximation). Maybe these tricks are effective enough, but that's overly empirical.\n* The experimental section is empirical and lacks theoretical analysis. Your analysis of the fMRI encoder with ROIs is interesting, but little focus had been put on what's the scientific relationship of neuroscience and diffusion model.\n\n\n\n[1] https://www.biorxiv.org/content/10.1101/2022.11.18.517004v3\n\n[2] https://arxiv.org/abs/2112.10752\n\n[3] https://arxiv.org/pdf/2208.01618.pdf\n\nQUESTIONS:\nPlease consider the things listed in the “Weaknesses” section.\n\nAlso please consider providing information regarding the interpretability of the utilization of the diffusion model in your context.\n\nLIMITATIONS:\nTo my knowledge and understanding, there is no potential negative societal impact of this work. Other limitations please see “Weaknesses”.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a new algorithm to guide a diffusion model to decode maximally activating images for particular voxel subregions of human fMRI using an encoding model trained to predict brain activity from images. The algorithm identifies stereotypic features for defined ROIs, such as faces or food items. The authors evaluate the specificity of the reconstructed image with CLIP zero-shot classification and human evaluation. They also cluster the encoding model weights and find that the clusters result in perceivable semantic categories in the reconstructed images. This algorithm allows for more fine-grained analysis of preferences across the visual system.\n\nSTRENGTHS:\nThere have been many papers that reconstruct images from brain activity. However, all of them require some form of retraining the diffusion model. This approach only needs an encoding model that can be trained independently. \n\nThe extensive human evaluation is great. While this goes beyond what I would ask for a NeurIPS paper, I think it would be great to also verify them by showing them back to human subjects.\n\nWEAKNESSES:\nThe authors show that the reconstructed images possess stable properties that can be identified by humans and by a CLIP network. However, they don't show that the images also do what they are supposed to do, which is activate a particular subnetwork. While human experiments probably go beyond the scope of this NeurIPS paper, one way to do that would be to take **another** encoding model for the same neural activity, take it as a proxy for the brain and show the images to that.\n\nQUESTIONS:\nQ1: As far as I understand $\\gamma$ trades off between the denoising gradient and the maximization gradient. I couldn't find what value you set it too, or how you chose it. \n\nQ2: There is a preprint that describes pretty much the same idea as in the paper but uses it for V4 single neurons:\n\nPawel A. Pierzchlewicz, Konstantin F. Willeke, Arne F. Nix, Pavithra Elumalai, Kelli Restivo, Tori Shinn, Cate Nealley, Gabrielle Rodriguez, Saumil Patel, Katrin Franke, Andreas S. Tolias, Fabian H. Sinz Energy Guided Diffusion for Generating Neurally Exciting Images\nhttps://www.biorxiv.org/content/10.1101/2023.05.18.541176v1\n\nIf one looks at the date, the preprint has been submitted around the NeurIPS deadline, so it's unlikely that the authors were aware of it and I would consider this as a case where two groups had the same idea. However, I think it would be fair to mention/cite them.  \n\nQ3: Can you discuss the compute time required for a single image?\n\nLIMITATIONS:\nLimitations are discussed.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis study proposes BrainDIVE, a system for synthesizing optimal stimuli for any given region of interest in the brain. The model combines a pretrained latent diffusion model for image generation with a linear “brain encoder” trained to map CLIP feature vectors onto the corresponding brain activity. At test time, a gradient-based optimization iteratively produces an image that maximizes the activity of a predefined set of voxels. The technique is validated on well-known ROIs, producing the expected images. It can also highlight subtle differences between ROIs selective to the same broad category, or functional subdivisions of existing ROIs. The functional hypotheses are validated by human subjects evaluating the properties of the generated pictures. The technique appears simple but can provide meaningful information for neuroscience studies.\n\nSTRENGTHS:\n* The method can retrieve subtle differences between ROIs selective for the same class (e.g. OFA and FFA).\n* The method can highlight functional differences between sub-clusters of existing ROIS (e.g. food clusters)\n* The qualitative observations are validated by behavioral evaluations from human subjects.\n\nWEAKNESSES:\n* The Related works section tends to overstate the novelty of the technique. It only mentions NeuroGen as prior work, whereas other prior studies had also attempted to generate optimal stimuli for specific ROIs: for instance, Ratan Murty et al (2021), Ozcelik et al (2022), Ozcelik et al (2023). The latter also used a diffusion model for image generation, as done here. Furthermore, you seem to be aware of at least some of these studies, since you criticized one of them in your Methods section for using a “hand-derived” prior. It would be much better to properly acknowledge all these studies in the Related works section, and clarify your method’s advantages/disadvantages (e.g. no hand-derived prior/time-consuming iterative method).\n\nQUESTIONS:\n* I do not understand why you systematically need to report the top-5 or top-10 out of 500 generated images? Do I understand correctly that the pool of 500 are all generated for the same objective (ROI maximization)? If yes, then what does this imply about the remaining 99% of images? Are they worse than NeuroGen? Are they actually failure cases? What do the worse 1% of images look like? If they are not representative of the expected category, does that invalidate your optimization method?\n* Appendix, section 9, training objective: this section is useful to understand the image optimization process. One thing I do not understand from this section or from Figure 2 is why the optimization is performed in the diffusion latent space, rather than in CLIP space? Why do the gradients have to flow all the way into the diffusion model? Couldn’t you iteratively optimize the $CLIP_{img}$ vector for the same objective, and then use it to condition the diffusion model in one pass (as you would with a text prompt)? This should be computationally more efficient, and I don’t see any reason why it would be less accurate.\n\nLIMITATIONS:\nLimitations are properly acknowledged.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Brain Diffusion for Visual Exploration (BrainDiVE) that aimed at exploring the fine-grained functional organization of the human visual cortex. Motivated by the limitations of previous studies that relied on researcher-crafted stimuli, BrainDiVE leverages generative deep learning models trained on large-scale image datasets and brain activation data from fMRI recordings. The proposed method uses brain maps as guidance to synthesize diverse and realistic images, enabling data-driven exploration of semantic preferences across visual cortical regions. By applying BrainDiVE to category-selective voxels and individual ROIs, the authors demonstrate its ability to capture semantic selectivity and identify subtle differences in response properties within specific brain networks. It is also shown that BrainDiVE identifies novel functional subdivisions within existing ROIs, highlighting its potential for providing new insights into the human visual system's functional properties.\n\nSTRENGTHS:\nThe paper's methodology shows promise in capturing semantic selectivity and identifying fine-grained functional distinctions within visual cortical regions. Also, the paper's potential significance lies in applying BrainDiVE to understand the fine-grained functional organisation of the human visual cortex. \nBy providing insights into category selectivity, response properties, and sub-regional divisions, the paper opens avenues for further exploratory neuroscience studies. \nTo achieve this, the authors perform the experiments in the manuscript are extensive, covering:\n1. the semantic specificity of the method by decoding images from task fMRI and literature-obtained ROIs\n2. compared the abstraction of face representation in the brain by comparing images decoded from two regions, the fusiform face area (FFA) and the occipital face area (OFA). \n3. Use the method to extend knowledge of brain function by finding subdivisions in known areas in the cases of food decoding.\nThe authors describe the experimental setting clearly at the beginning of section 4 for all cases.\nThe experiments show that the manuscript has a good balance between the combination of known methodological approaches and addressing interesting questions in neuroscience. Specifically, the authors perform a commendable effort in obtaining quantitative results using human evaluators (cf Tables 3 and 4).\nFinally, the authors show through results evidence that their method is a window into understanding region-specificity hypotheses of brain regions which are knowingly involved in different aspects of visual processing.\n\nWEAKNESSES:\nThe paper could benefit from comparing its results with previous contributions [e.g. 9 and 10] to highlight its novelty and contributions in relation to existing methods. \nThe methodology part lacks clarity, particularly in explaining key components like the diffusion model architecture and brain-guided synthesis—further clarifying the role of the image-to-brain encoder in influencing the denoising process during inference.\nThe generalisability and reliability of results are hard to asses through a small dataset, specifically just 10 subjects. \nFurthermore, the paper lacks clarity on how the sub-divisions of the visual cortex are being verified or validated. Whether the results are specific to the analysed regions or the algorithm is being too biased by the experimental condition is not clear through the experiments. For instance, what would happen if the authors try to decode an area not specific to visual processing? An example of this would be using subsections of the orbitofrontal gyrus or other brain areas not expected to perform well in reconstructing visual stimuli.\n\nAs a small point authors should review the presentation of images. Figure 1 shows mostly best-case scenarios which are then not as good in Figure 4 (for instance the case of images generated from face voxels). This might bias readers. Second, some details, such as figure 4 appearing before figure 3 make reading the paper confusing.\n\nQUESTIONS:\n1. Validation of BrainDiVE's Effectiveness:\n  a. Can the authors provide more details on the validation process to ensure that BrainDiVE-generated images indeed effectively activate the targeted brain regions? \n  b. Could the authors consider conducting more extensive comparisons with other existing methods [e.g. 9 and 10], to demonstrate the advantages and uniqueness of BrainDiVE in eliciting specific brain activations?\n  c. Could the authors show the results on decoding a baseline region which is not involved in visual processing?\n\n2. Clarity in Methodology: The methodology part could benefit from more clarity and detailed explanations of key components, such as the diffusion model architecture and the exact implementation of brain-guided synthesis. Improving the architecture in Figure-2 might help.\n\n3. Statistical Analysis of Qualitative Evaluation: As the paper relies on qualitative evaluation with 10 subjects, could the authors mention this explicitly and what limitations are expected from this in the limitations section?\n\n4. Ethical Considerations: Could the authors describe the demographics of the population, or acknowledge the lack of this information to inform of wether the results are biased to a specific gender or population group.\n\n5. Reproducibility: Are the authors intending to release the code and not publicly available data such as the scores produced by the human raters upon acceptance of the paper? This is key to guarantee reproducibility\n\nLIMITATIONS:\nAuthors have not addressed the negative societal impact of their work, but this can be fixed by adding specific text in the Discussion section. The authors don't mention the demographic characteristics of the small human subject database that they have used not mention any ethical concerns related to decoding images from brain activations. This is fixable so authors should address it.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-21"}