SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Andrew F. Luo
afluo@cmu.edu
Margaret M. Henderson
mmhender@cmu.edu
Leila Wehbe*
lwehbe@cmu.edu
Michael J. Tarr*
michaeltarr@cmu.edu
A long standing goal in neuroscience has been to elucidate the functional organization of the brain. Within higher visual cortex, functional accounts have remained relatively coarse, focusing on regions of interest (ROIs) and taking the form of selectivity for broad categories such as faces, places, bodies, food, or words. Because the identification of such ROIs has typically relied on manually assembled stimulus sets consisting of isolated objects in non-ecological contexts, exploring functional organization without robust a priori hypotheses has been challenging. To overcome these limitations, we introduce a data-driven approach in which we synthesize images predicted to activate a given brain region using paired natural images and fMRI recordings, bypassing the need for category-specific stimuli. Our approach – Brain Diffusion for Visual Exploration ("BrainDiVE") – builds on recent generative methods by combining large-scale diffusion models with brain-guided image synthesis. Validating our method, we demonstrate the ability to synthesize preferred images with appropriate semantic specificity for well-characterized category-selective ROIs. We then show that BrainDiVE can characterize differences between ROIs selective for the same high-level category. Finally we identify novel functional subdivisions within these ROIs, validated with behavioral data. These results advance our understanding of the fine-grained functional organization of human visual cortex, and provide well-specified constraints for further examination of cortical organization using hypothesis-driven methods.
Code and project site: https://www.cs.cmu.edu/~afluo/BrainDiVE
The human visual cortex plays a fundamental role in our ability to process, interpret, and act on visual information. While previous studies have provided important evidence that regions in the higher visual cortex preferentially process complex semantic categories such as faces, places, bodies, words, and food [1, 2, 3, 4, 5, 6, 7], these important discoveries have been primarily achieved through the use of researcher-crafted stimuli. However, hand-selected, synthetic stimuli may bias the results or may not accurately capture the complexity and variability of natural scenes, sometimes leading to debates about the interpretation and validity of identified functional regions [8]. Furthermore, mapping selectivity based on responses to a fixed set of stimuli is necessarily limited, in that it can only identify selectivity for the stimulus properties that are sampled. For these reasons, data-driven methods for interpreting high-dimensional neural tuning are complementary to traditional approaches.
We introduce Brain Diffusion for Visual Exploration ("BrainDiVE"), a generative approach for synthesizing images that are predicted to activate a given region in the human visual cortex. Several
* Co-corresponding Authors
Face regions
Figure 1: Images generated using BrainDive. Images are generated using a diffusion model with maximization of voxels identified from functional localizer experiments as conditioning. We find that brain signals recorded via fMRI can guide the synthesis of images with high semantic specificity, strengthening the evidence for previously identified category selective regions. Select images are shown, please see below for uncurated images.
Place regions
Body regions
Word regions
Food regions
recent studies have yielded intriguing results by combining deep generative models with brain guidance [9, 10, 11]. BrainDiVE, enabled by the recent availability of large-scale fMRI datasets based on natural scene images [12, 13], allows us to further leverage state-of-the-art diffusion models in identifying fine-grained functional specialization in an objective and data-driven manner. BrainDiVE is based on image diffusion models which are typically driven by text prompts in order to generate synthetic stimuli [14]. We replace these prompts with maximization of voxels in given brain areas. The result being that the resultant synthesized images are tailored to targeted regions in higher-order visual areas. Analysis of these images enables data-driven exploration of the underlying feature preferences for different visual cortical sub-regions. Importantly, because the synthesized images are optimized to maximize the response of a given sub-region, these images emphasize and isolate critical feature preferences beyond what was present in the original stimulus images used in collecting the brain data. To validate our findings, we further performed several human behavioral studies that confirmed the semantic identities of our synthesized images.
More broadly, we establish that BrainDiVE can synthesize novel images (Figure 1) for category-selective brain regions with high semantic specificity. Importantly, we further show that BrainDiVE can identify ROI-wise differences in selectivity that map to ecologically relevant properties. Building on this result, we are able to identify novel functional distinctions within sub-regions of existing ROIs. Such results demonstrate that BrainDiVE can be used in a data-driven manner to enable new insights into the fine-grained functional organization of the human visual cortex.
Mapping High-Level Selectivity in the Visual Cortex. Certain regions within the higher visual cortex are believed to specialize in distinct aspects of visual processing, such as the perception of faces, places, bodies, food, and words [15, 3, 4, 1, 16, 17, 18, 19, 5, 20]. Many of these discoveries rely on carefully handcrafted stimuli specifically designed to activate targeted regions. However, activity under natural viewing conditions is known to be different [21]. Recent efforts using artificial neural networks as image-computable encoders/predictors of the visual pathway [22, 23, 24, 25, 26, 27, 28, 29, 30] have facilitated the use of more naturalistic stimulus sets. Our proposed method incorporates an image-computable encoding model in line with this past work.
Deep Generative Models. The recent rise of learned generative models has enabled sampling from complex high dimensional distributions. Notable approaches include variational autoencoders [31, 32], generative adversarial networks [33], flows [34, 35], and score/energy/diffusion models [36, 37, 38, 39]. It is possible to condition the model on category [40, 41], text [42, 43], or images [44]. Recent diffusion models have been conditioned with brain activations to reconstruct observed images [45, 46, 47, 48, 49]. Unlike BrainDiVE, these approaches tackle reconstruction but not synthesis of novel images that are predicted to activate regions of the brain.
Brain-Conditioned Image Generation. The differentiable nature of deep encoding models inspired work to create images from brain gradients in mice, macaques, and humans [50, 51, 52]. Without constraints, the images recovered are not naturalistic. Other approaches have combined deep generative models with optimization to recover natural images in macaque and humans [10, 11, 9]. Both [11, 9] utilize fMRI brain gradients combined with ImageNet trained BigGAN. In particular [11] performs end-to-end differentiable optimization by assuming a soft relaxation over the 1,000 ImageNet classes; while [9] trains an encoder on the NSD dataset [13] and first searches for
top-classes, then performs gradient optimization within the identified classes. Both approaches are restricted to ImageNet images, which are primarily images of single objects. Our work presents major improvements by enabling the use of diffusion models [44] trained on internet-scale datasets [53] over three magnitudes larger than ImageNet. Concurrent work by [54] explore the use of gradients from macaque V4 with diffusion models, however their approach focuses on early visual cortex with grayscale image outputs, while our work focuses on higher-order visual areas and synthesize complex compositional scenes. By avoiding the search-based optimization procedures used in [9], our work is not restricted to images within a fixed class in ImageNet. Further, to the authors' knowledge we are the first work to use image synthesis methods in the identification of functional specialization in sub-parts of ROIs.
Figure 2: Architecture of brain guided diffusion (BrainDiVE). Top: Our framework consists of two core components: (1) A diffusion model trained to synthesize natural images by iterative denoising; we utilize pretrained LDMs. (2) An encoder trained to map from images to cortical activity. Our framework can synthesize images that are predicted to activate any subset of voxels. Shown here are scene-selective regions (RSC/PPA/OPA) on the right hemisphere. Bottom: We visualize every 4 steps the magnitude of the gradient of the brain w.r.t. the latent and the corresponding "predicted  $x_0$ " [55] when targeting scene selective voxels in both hemispheres. We find clear structure emerges.
We aim to generate stimuli that maximally activate a given region in visual cortex using paired natural image stimuli and fMRI recordings. We first review relevant background information on diffusion models. We then describe how we can parameterize encoding models that map from images to brain data. Finally, we describe how our framework (Figure 2) can leverage brain signals as guidance to diffusion models to synthesize images that activate a target brain region.
3.1 Background on Diffusion Models
Diffusion models enable sampling from a data distribution  $p(x)$  by iterative denoising. The sampling process starts with  $x_{T} \sim \mathcal{N}(0,\mathbb{I})$ , and produces progressively denoised samples  $x_{T - 1}, x_{T - 2}, x_{T - 3} \ldots$  until a sample  $x_0$  from the target distribution is reached. The noise level varies by timestep  $t$ , where the sample at each timestep is a weighted combination of  $x_0$  and  $\epsilon \sim \mathcal{N}(0,\mathbb{I})$ , with  $x_{t} = \sqrt{\alpha_{t}} x_{0} + \epsilon \sqrt{1 - \alpha_{t}}$ . The value of  $\alpha$  interpolates between  $\mathcal{N}(0,\mathbb{I})$  and  $p(x)$ .
In the noise prediction setting, an autoencoder network  $\epsilon_{\theta}(x_t,t)$  is trained using a mean-squared error  $\mathbb{E}_{(x,\epsilon ,t)}\left[\| \epsilon_{\theta}(x_t,t) - \epsilon \| _2^2\right]$ . In practice, we utilize a pretrained latent diffusion model (LDM) [44], with learned image encoder  $E_{\Phi}$  and decoder  $D_{\Omega}$ , which together act as an autoencoder  $\mathcal{I} \approx D_{\Omega}(E_{\Phi}(\mathcal{I}))$ . The diffusion model is trained to sample  $x_0$  from the latent space of  $E_{\Phi}$ .
3.2 Brain-Encoding Model Construction
A learned voxel-wise brain encoding model is a function  $M_{\theta}$  that maps an image  $\mathcal{I} \in \mathbb{R}^{3 \times H \times W}$  to the corresponding brain activation fMRI beta values represented as an  $N$  element vector  $B \in \mathbb{R}^N$ :  $M_{\theta}(\mathcal{I}) \Rightarrow B$ . Past work has identified later layers in neural networks as the best predictors of higher visual cortex [30, 56], with CLIP trained networks among the highest performing brain
encoders [28, 57]. As our target is the higher visual cortex, we utilize a two component design for our encoder. The first component consists of a CLIP trained image encoder which outputs a  $K$  dimensional vector as the latent embedding. The second component is a linear adaptation layer  $W \in \mathcal{R}^{N \times K}$ ,  $b \in \mathcal{R}^N$ , which maps euclidean normalized image embeddings to brain activation.
$$
B \approx M _ {\theta} (\mathcal {I}) = W \times \frac {\operatorname {C L I P} _ {\text {i m g}} (\mathcal {I})}{\| \operatorname {C L I P} _ {\text {i m g}} (\mathcal {I}) \| _ {2}} + b
$$
Optimal  $W^{*}, b^{*}$  are found by optimizing the mean squared error loss over images. We observe that use of a normalized CLIP embedding improves stability of gradient magnitudes w.r.t. the image.
3.3 Brain-Guided Diffusion Model
BrainDiVE seeks to generate images conditioned on maximizing brain activation in a given region. In conventional text-conditioned diffusion models, the conditioning is done in one of two ways. The first approach modifies the function  $\epsilon_{\theta}$  to further accept a conditioning vector  $c$ , resulting in  $\epsilon_{\theta}(x_t,t,c)$ . The second approach uses a contrastive trained image-to-concept encoder, and seeks to maximize a similarity measure with a text-to-concept encoder.
Conditioning on activation of a brain region using the first approach presents difficulties. We do not know a priori the distribution of other non-targeted regions in the brain when a target region is maximized. Overcoming this problem requires us to either have a prior  $p(B)$  that captures the joint distribution for all voxels in the brain, to ignore the joint distribution that can result in catastrophic effects, or to use a handcrafted prior that may be incorrect [47]. Instead, we propose to condition the diffusion model via our image-to-brain encoder. During inference we perturb the denoising process using the gradient of the brain encoder maximization objective, where  $\gamma$  is a scale, and  $S \subseteq N$  are the set of voxels used for guidance. We seek to maximize the average activation of  $S$  predicted by  $M_{\theta}$ :
$$
\epsilon_ {t h e t a} ^ {\prime} = \epsilon_ {t h e t a} - \sqrt {1 - \alpha_ {t}} \nabla_ {x _ {t}} (\frac {\gamma}{| S |} \sum_ {i \in S} M _ {\theta} (D _ {\Omega} (x _ {t} ^ {\prime})) _ {i})
$$
Like [14, 58, 59], we observe that convergence using the current denoised  $x_{t}$  is poor without changes to the guidance. This is because the current image (latent) is high noise and may lie outside of the natural image distribution. We instead use a weighted reformulation with an euler approximation [55, 59] of the final image:
$$
\hat {x} _ {0} = \frac {1}{\sqrt {\alpha}} (x _ {t} - \sqrt {1 - \alpha} \epsilon_ {t})
$$
$$
x _ {t} ^ {\prime} = (\sqrt {1 - \alpha}) \hat {x} _ {0} + (1 - \sqrt {1 - \alpha}) x _ {t}
$$
By combining an image diffusion model with a differentiable encoding model of the brain, we are able to generate images that seek to maximize activation for any given brain region.
In this section, we use BrainDiVE to highlight the semantic selectivity of pre-identified category-selective voxels. We then show that our model can capture subtle differences in response properties between ROIs belonging to the same broad category-selective network. Finally, we utilize BrainDiVE to target finer-grained sub-regions within existing ROIs, and show consistent divisions based on semantic and visual properties. We quantify these differences in selectivity across regions using human perceptual studies, which confirm that BrainDiVE images can highlight differences in tuning properties. These results demonstrate how BrainDiVE can elucidate the functional properties of human cortical populations, making it a promising tool for exploratory neuroscience.
We utilize the Natural Scenes Dataset (NSD; [13]), which consists of whole-brain 7T fMRI data from 8 human subjects, 4 of whom viewed 10,000 natural scene images repeated  $3 \times$ . These subjects, S1, S2, S5, and S7, are used for analyses in the main paper (see Supplemental for results for additional subjects). All images are from the MS COCO dataset. We use beta-weights (activations) computed using GLMSingle [60] and further normalize each voxel to  $\mu = 0$ ,  $\sigma = 1$  on a per-session basis. We average the fMRI activation across repeats of the same image within a subject. The  $\sim 9,000$  unique images for each subject ([13]) are used to train the brain encoder for each subject, with the remaining  $\sim 1,000$  shared images used to evaluate  $R^2$ . Image generation is on a per-subject basis and done
on an Nvidia V100 using 1,500 compute hours. As the original category ROIs in NSD are very generous, we utilize a stricter  $t > 2$  threshold to reduce overlap unless otherwise noted. The final category and ROI masks used in our experiments are derived from the logical AND of the official NSD masks with the masks derived from the official  $t$ -statistics.
We utilize stable-diffusion-2-1-base, which produces images of  $512 \times 512$  resolution using  $\epsilon$ -prediction. Following best practices, we use multi-step 2nd order DPM-Solver++ [61] with 50 steps and apply 0.75 SAG [62]. We set step size hyperparameter  $\gamma = 130.0$ . Images are resized to  $224 \times 224$  for the brain encoder. (" (null prompt) is used as the input prompt, thus the diffusion performs unconditional generation without brain guidance. For the brain encoder we use ViT-B/16, for CLIP probes we use CoCa ViT-L/14. These are the highest performing LAION-2B models of a given size provided by OpenCLIP [63, 64, 65, 66]. We train our brain encoders on each human subject separately to predict the activation of all higher visual cortex voxels. See Supplemental for visualization of test time brain encoder  $R^2$ . To compare images from different ROIs and sub-regions (OFA/FFA in 4.3, two clusters in 4.4), we asked human evaluators select which of two image groups scored higher on various attributes. We used 100 images from each group randomly split into 10 non-overlapping subgroups. Each human evaluator performed 80 comparisons, across 10 splits, 4 NSD subjects, and for both fMRI and generated images. See Supplemental for standard error of responses. Human evaluators provided written informed consent and were compensated at $12.00/hour. The study protocol was approved by the institutional review board at the authors' institution.
4.2 Broad Category-Selective Networks
In this experiment, we target large groups of category-selective voxels which can encompass more than one ROI (Figure 3). These regions have been previously identified as selective for broad semantic categories, and this experiment validates our method using these identified regions. The face-, place-, body-, and word- selective ROIs are identified with standard localizer stimuli [67]. The food-selective voxels were obtained from [5].
The same voxels were used to select the top guide the generation of BrainDiVE images.
Figure 3: Visualizing category-selective voxels in S1. See text for details on how category selectivity was defined.
In Figures 4 we visualize, for place-, face-, word-, and body- selective voxels, the top-5 out of 10,000 images from the fMRI stimulus set (NSD), and the top-5 images out of 1,000 total images as evaluated by the encoding component of BrainDiVE. For food selective voxels, the top-10 are visualized. A visual inspection indicates that our method is able to generate diverse images that semantically represent the target category. We further use CLIP to perform semantic probing of the images, and force the images to be classified into one of five categories. We measure the percentage of images that match the preferred category for a given set of voxels (Table 1). We find that our top-  $10\%$  and  $20\%$  of images exceed the top-  $1\%$  and  $2\%$  of natural images in accuracy, indicating our method has high semantic specificity.
Faces	Places	Bodies	Words	Food	Mean
S1↑	S2↑	S1↑	S2↑	S1↑	S2↑	S1↑	S2↑	S1↑	S2↑	S1↑	S2↑
NSD all stim	17.4	17.2	29.9	29.5	31.6	31.8	10.3	10.6	10.8	10.9	20.0	20.0
NSD top-200	42.5	41.5	66.5	80.0	56.0	65.0	31.5	34.5	68.0	85.5	52.9	61.3
NSD top-100	40.0	45.0	68.0	79.0	49.0	60.0	30.0	49.0	78.0	85.0	53.0	63.6
BrainDiVE-200	69.5	70.0	97.5	100	75.5	68.5	60.0	57.5	89.0	94.0	78.3	75.8
BrainDiVE-100	61.0	68.0	97.0	100	75.0	69.0	60.0	62.0	92.0	95.0	77.0	78.8
Table 1: Evaluating semantic specificity with zero-shot CLIP classification. We use CLIP to classify images from each ROI into five semantic categories: face/place/body/word/food. Shown is the percentage where the classified category of the image matches the preferred category of the brain region. We show this for each subject's entire NSD stimulus set (10,000 images for S1&S2); the top-200 and top-100 images (top-2% and top-1%) evaluated by mean true fMRI beta, and the top-200 and top-100 (20% and 10%) of BrainDiVE images as self-evaluated by the encoding component of BrainDiVE. BrainDiVE generates images with higher semantic specificity than the top 1% of natural images for each brain region.
Figure 4: Results for category selective voxels (S1). We identify the top-5 images from the stimulus set or generated by our method with highest average activation in each set of category selective voxels for the face/place/word/body categories, and the top-10 images for the food selective voxels.
Which ROI has more...	photorealistic faces	animals	abstract shapes/lines
S1	S2	S5	S7	S1	S2	S5	S7	S1	S2	S5	S7
FFA-NSD	45	43	34	41	34	34	17	15	21	6	14	22
OFA-NSD	25	22	21	18	47	36	65	65	24	44	28	25
FFA-BrainDiVE	79	89	60	52	17	13	21	19	6	11	18	20
OFA-BrainDiVE	11	4	15	22	71	61	52	50	80	79	40	39
Table 2: Human evaluation of the difference between face-selective ROIs. Evaluators compare groups of images corresponding to OFA and FFA; comparisons are done within GT and generated images respectively. Questions are posed as: "Which group of images has more X?"; options are FFA/OFA/Same. Results are in %. Note that the "Same" responses are not shown; responses across all three options sum to 100.
4.3 Individual ROIs
In this section, we apply our method to individual ROIs that are selective for the same broad semantic category. We focus on the occipital face area (OFA) and fusiform face area (FFA), as initial tests suggested little differentiation between ROIs within the place-, word-, and body- selective networks. In this experiment, we also compare our results against the top images for FFA and OFA from NeuroGen [9], using the top 100 out of 500 images provided by the authors. Following NeuroGen, we also generate 500 total images, targeting FFA and OFA separately (Figure 5). We observe that both diffusion-generated and NSD images have very high face content in FFA, whereas NeuroGen has higher animal face content. In OFA, we observe both NSD and BrainDiVE images have a strong face component, although we also observe text selectivity in S2 and animal face selectivity in S5. Again NeuroGen predicts a higher animal component than face for S5. By avoiding the use of fixed categories, BrainDiVE images are more diverse than those of NeuroGen. This trend of face and animals appears at  $t > 2$  and the much stricter  $t > 5$  threshold for identifying face-selective voxels ( $t > 5$  used for visualization/evaluation). The differences in images synthesized by BrainDiVE for FFA and OFA are consistent with past work suggesting that FFA represents faces at a higher level of abstraction than OFA, while OFA shows greater selectivity to low-level face features and sub-components, which could explain its activation by off-target categories [68, 69, 70].
To quantify these results, we perform a human study where subjects are asked to compare the top-100 images between FFA & OFA, for both NSD and generated images. Results are shown in Table 2.
Figure 5: Results for face-selective ROIs. For each ROI (OFA, FFA) we visualize the top-5 images from NSD and NeuroGen, and the top-10 from BrainDiVE. NSD images are selected using the fMRI betas averaged within each ROI. NeuroGen images are ranked according to their official predicted ROI activity means. BrainDiVE images are ranked using our predicted ROI activities from 500 images. Red outlines in the NSD images indicate examples of responsiveness to non-face content.
We find that OFA consistently has higher animal and abstract content than FFA. Most notably, this difference is on average more pronounced in the images from BrainDiVE, indicating that our approach is able to highlight subtle differences in semantic selectivity across regions.
Figure 6: Clustering within the food ROI and within OPA. Clustering of encoder model weights for each region is shown for two example subjects on an inflated cortical surface.
4.4 Semantic Divisions within ROIs