BrainSCUBA: Fine-Grained Natural Language Captions of Visual Cortex Selectivity
Authors: Andrew Luo
Margaret Marie Henderson
Michael J. Tarr
Leila Wehbe
Venue: ICLR 2024 poster (2024)
Keywords: neuroscience
brain
fmri
captions
image synthesis
visual cortex
computational neuroscience
Primary area: applications to neuroscience & cognitive science
TL;DR: BrainSCUBA: Fine-Grained Natural Language Captions of Visual Cortex Selectivity

Abstract: Understanding the functional organization of higher visual cortex is a central focus in neuroscience. Past studies have primarily mapped the visual and semantic selectivity of neural populations using hand-selected stimuli, which may potentially bias results towards pre-existing hypotheses of visual cortex functionality. Moving beyond conventional approaches, we introduce a data-driven method that generates natural language descriptions for images predicted to maximally activate individual voxels of interest. Our method -- Semantic Captioning Using Brain Alignments ("BrainSCUBA") -- builds upon the rich embedding space learned by a contrastive vision-language model and utilizes a pre-trained large language model to generate interpretable captions. We validate our method through fine-grained voxel-level captioning across higher-order visual regions. We further perform text-conditioned image synthesis with the captions, and show that our images are semantically coherent and yield high predicted activations. Finally, to demonstrate how our method enables scientific discovery, we perform exploratory investigations on the distribution of "person" representations in the brain, and discover fine-grained semantic selectivity in body-selective areas. Unlike earlier studies that decode text, our method derives *voxel-wise captions of semantic selectivity*. Our results show that BrainSCUBA is a promising means for understanding functional preferences in the brain, and provides motivation for further hypothesis-driven investigation of visual cortex.

Published as a conference paper at ICLR 2024
BRAINSCUBA: FINE-GRAINED NATURAL LANGUAGE CAPTIONS OF VISUAL CORTEX SELECTIVITY
Andrew Luo Carnegie Mellon University afluo @ cmu.edu
Margaret M. Henderson Carnegie Mellon University
mmbender@ @ @cmueddu
Michael J. Tarr Carnegie Mellon University michaeltarr @mmuedu
Leila Wehbe
Carnegie Mellon University Iwehbe @ cmm.edu
ABSTRACT
Understanding the functional organization of higher visual cortex is central focus in neuroscience. Past studies have primarily mapped the visual and semantic selectivity of neural populations using hand-selected stimuli, which may potentially bias results towards pre-existing hypotheses of visual cortex functionality. Moving beyond conventional approaches, we introduce data-driven method that generates natural language descriptions for images predicted maximally activate individual voxels of interest. Our method Semantic Captioning Using Brain Alignments ("BrainSCUBBA") builds upon the rich embedding space learned by contrastive Vision-language model and utilizes a pre-trained large language model to generate interpretable captions. We validate our method through fine-graine' voxel-level captioning across higher-order visual regions. We further perform text-conditioned image synthesis with the captions, and show that images semantically coherent and yield high predicted activations. Finally, to demonstrate how our method enables scientific discovery, we perform exploratory investigations the distribution of "person" representations in the brain, and discover fine-grained semantic selectivity in body-selective areas. Unlike earlier studies that decode text, our method derives voxel-wise captions of semantic selectivity. Our results show that BrainSCUBA a promising means for understanding functional preferences in the brain, and provides motivation for further hypothesis-driven investigation of visual cortex. Code and project site: https///www.cs.cmm. wttp./cmmm.ddaaaa///aaaaaaaa Brans/BrainSCUBA
INTRODUCTION
The recognition of complex objects and semantic visual concepts is supported by a network of regions within higher visual cortex. Past research has identified the specialization of certain regions in processing semantic categories such as faces, places, bodies, words, and food (Puce al., 1996; Kanwisher et al., 1997; McCarthy et al., 1997; Maguire, 2001; Epstein & Kanwisher, 1998; Grill- Spector, 2003; Downing et al., 2001; Khosla al., 2022; Pennock al., 2023; Jain al., 2023). Notably, the discovery of these regions has largely relied on a hypothesis-driven approach, whereby the researcher hand-selects stimuli study specific hypothesis. This approach risk biasing the results as it may fail to capture the complexity and variability inherent in real-world images, which can lead to disagreements regarding region's functional selectivity (Gauthier et al., 1999).
To better address these issues, we introduce BrainSCUBA (Semantic Captioning Using Brain Alignments), an approach for synthesizing per-voxel natural language captions that describe voxel- wise preferred stimuli. Our method builds upon the availability of large-scale fMRI datasets (Allen et al., 2022) with a natural image viewing task, and allows us to leverage contrastive vision-language models and large-language models in identifying fine-grained voxel-wise functional specialization in data-driven manner. BrainSCUBA is conditioned on weights from an conge--bomutablee fMRI encoder that maps from image to voxel-wise brain activations. The design of our encoder allows us to extract the optimal encoder embedding for each voxel, and we use a training- -free method to close the modality gap between the encoder-weige weight space and natural images. The output of Brain

---

Published as a conference paper at ICLR 2024
describes (in words) the visual stimulus that maximally activates a given voxel. Interpretation and visualization of these captions facilitates data-driven investigation into the underlying feature preferences across various visual sub-regions in the brain.
In contrast to earlier studies that decode text from the brain activity related to an image, we demon- strate voxel-wise captions of semantic selectivity. Concretely, we show that method captures the categorical selectivity of multiple regions in visual cortex. Critically, the content of the captions replicates the field's pre-existing knowledge of each region's 's preferred category. We further show that BrainSCUBA combined with a texi-to-image model can generate images semantically aligned with targeted brain regions and yield high predicted activations when evaluated with a different encoder backbone. Finally, we use BrainSCUBA to perform data-driven exploration for the coding of the category "person", finding evidence for person-selective regions outside of the commonly recognized face/body domains and discovering new finer-grained selectivity within known body-selective areas.
2 RELATED WORK
Several recent studies have yielded intriguing results by using large-scale vision-language models to reconstruct images and text-descriptions from brain patterns when viewing images (Takagi & Nishi- moto, 2022; Chen et al., 2022; Doerig et al., 2022; Ferrante et al., 2023; Ozcelik & VanRullen, 2023; Liu et al., 2023), or to generate novel images that are predicted to activate a given region (Ratan Murty al., 2021; Gu et al., 2022; Luo et al., 2023). Broadly speaking, these approaches require condition- ing on broad regions of the visual cortex, and have not demonstrated the ability to scale down and enable voxel-level understanding of neural selectivity. Additionally, these methods produce images rather than interpretable captions. Work on artificial neurons (Borowski et al., 2020; Zimmermann al., 2021) have shown that feature visualization may not be more informative than top images in artificial neural networks. In contrast, our work tackles biological networks which have more noisy top-images that are less conducive to direct analysis, and the synthesis of novel images/captions can act as source of stimuli for future hypothesis-drive driven neuroscience studies.
Semantic Selectivity in Higher Visual Cortex. Higher visual cortex in the human brain contains regions which respond selectively to specific categories of visual stimuli, such as faces, places, bodies, words, and food (Desimone al., 1984; Puce al., 1996; Kanwisher et al., 1997; McCarthy et al., 1997; Maguire, 2001; Epstein & Kanwisher, 1998; Grill-Spector, 2003; Downing et al., 2001; Cohen et al., 2000; Khosla et al., 2022; Pennock et al., 2023; Jain et al., 2023). These discoveries have predominantly relied the use of hand-selected stimuli designed to trigger responses of distinct regions. However the handcrafted nature of these stimuli may misrepresent the complexity and diversity of visual information encountered in natural settings (Gallant et al., 1998; Felsen & Dan, 2005). In contrast, the recent progress in fMRI encoders that map from stimulus to brain response have enabled data-driven computational tests of brain selectivity in vision (Naselaris et al., 201 I; Huth et al., 2012; Yamins et al., 2014; Eickenberg et al., 2017; Wen et al., 2018; Kubilius et al., 2019; Conwell et al., 2023; Wang et al., 2022), language (Huth al., 2016; Deniz et al., 2019), and at the interface of vision and language (Popham et al., 2021). Here, based on Conwell et al. (2023)'s 's evaluation of the brain alignment of various pre-trained image models, we employ CLIP as our encoder backbone.
Image-Captioning with CLIP and Language Models. Vision-language models trained with a contrastive loss demonstrate remarkable capability across many discriminative tasks (Radford et al., 2021; Cherti al., 2023; Sun et al., 2023). However, due to the lack of a text-decoder, these models are typically paired with an adapted language model in order to produce captions. When captioning, some models utilize the full spatial CLIP embedding (Shen et al., 2021; Li et al., 2023a), whilst others use only the vector embedding (Mokady et al., 2021; Tewel et al., 2022; Li et al., 2023b). By leveraging the multi-modal latent space learned by CLIP, we are able to generate voxel-wise captions without
voxel-caption data.
Brain- Conditioned Image and Caption Generation. There are two broad directions when comes to brain conditioned generative models for vision. The first seeks to decode (reconstruct) visual inputs from the corresponding brain activations, including works that leverage retrieval, variational autoencoders (VAEs), generative adversarial networks (GANs), and scorr/energy/diiftsioon models (Kamitani & Tong. 2005; Han et al., 2019; Seeliger et al., 2018; Shen et al., 2019; Ren et al., 2021; Takagi & Nishimoto, 2022; Chen al., 2023; Lu al., 2023; Ozcelik & VanRullen, 2023).
2

---

Published as a conference paper at ICLR 2024
(a)
Voxel-wise linear adapter
CLIP
Visual stimuli
Frozen
Trained
Brain fMRI
(b)
Voxel-wise Weights
CLIP dim
orig
A close up of food on a plate on a toble
Single
W
Text
voxel weight
Decoder
Number of voxels
CLIP embeddings
Text
in brain
for natural images
(c) close up of person
An aerial view of bridge and a mountain
A group of people standing in room
A clase of clack on surface
plate of food on table
wearing a suit and tie
picture man and baby
A room that has a sink and window
group people sitting ground
Two men and woman holding up signs
close up cake on table
woman pose for picture Faces
garden
standing of sunset
up jar
display food items on table
benches and chairs
name it
Places
Bodies
Words
Food
Figure 1: Architecture of BrainSCUBA. (a) Our framework relies
fMRI encoder trained
to map from images to voxel-wise brain activations. The encoder consists of a frozen CLIP image network with a unit norm output and a linear probe. (b) We decode the voxel-wise weights by projecting the weights into the space of CLIP embeddings for natural images followed by sentence generation. (c) Select sentences from each region, please see experiments for a full analysis.
Some approaches further utilize generate captions that describe the observed visual stimuli (Doerig et al., 2022; Ferrante et al., 2023; Liu et al., 2023; Mai & Zhang, 2023).
The second approach seeks generate stimuli that activates a given region rather than exactly reconstructing the input (Walker et al., 2019; Bashivan et al., 2019). Some of these approaches utilize GANs Diffusion models constrain the synthesized output (Ponce al., 2019; Ratan Murty et al., 2021; Gu et al., 2022; Luo et al., 2023). BrainSCUBA falls under the broad umbrella of this second approach. But unlike prior methods which were restricted to modeling broad swathes of brain, our method can be applied at voxel-level, and can output concrete interpretable captions.
3 METHODS
We aim to generate fine-grained (voxel-level) natural language captions that describe a visual scene which maximally activate given voxel. We first describe the parameterization and training of voxel-wise fMRI encoder which goes from images to brain activations. We then describe how we can analytically derive the optimal CLIP embedding given the encoder weights. Finally, we describe how we close the gap between optimal CLIP embeddings and the natural image embedding space to enable excl-conditioned caption generation. We illustrate our framework Figure
3.1 MAAGI-TO-BRAIN ENCODER CONSTRUCTION
image-computable brain encoder is learned function Fg that transforms an image I Hxxx3 to voxel-wise brain activation beta values represented as a 1D vector of N brain voxels B E RI where (I) + B. Recent work identified models trained with a contrastive vision-language objective the highest performing feature extractor for visual cortex, with later CLIP layers being more accurate for higher visual areas (Wang et al., 2022; Conwell et al., 2023). As we seek to solely model higher-order visual areas, utilize two part design for encoder. First is frozen CLIP (Radford et al., 2021) backbone which outputs a R dimensional embedding vector for each image. The second is linear probe W MMN with bias E RIXN which transform unit-norm

---

Published as a conference paper at ICLR 2024
(a)
(b)
(c)
0.20
Images
simllariyy
(MAI encoder propprrjettonn
0.36
(MAI encoder projection
0.34
UMAP2
0.30
107
103
10 Number of vectors
10r
Figure 2: Projection of fMRI encoder weights. (a) We validate the encoder R2 test set, and find it can achieve high accuracy in the higher visual cortex. (b) The joint-UMAP of image CLIP embeddings, and pee-/posst-proectioon of the encoder. All embeddings are normalized before UMAP. (c) We measure the average cosine similarity between pre-/post-projection weights, and find increases as the images used are increased. Standard deviation of projections shown in light blue.
image embedding brain activations.
CLIP I
W +
+ B
(I)
[CLIPimg img l2)
After training with MSE loss, we evaluate the encoder on the test set in Figure 2(a) and find that our encoder can achieve high R².
3.2 DERIVING THE OPTIMAL EMBEDDING AND CLOSING THE GAP
The fMRI encoder we construct utilizes a linear probe applied to a unit-norm CLIP embedding. It follows from the design that the maximizing embedding e, for a voxel i can be derived efficiently from the weight, and the predicted activation is upper bounded by ||a ||2 + when
W
(2)
[ICLIP Pm (2-) et does not typically exist. There is modality gap between the CLIP embeddings of natural images and the optimal embedding derived from the linear weight matrix. We visualize this gap in Figure 2(b) in joint UMAP (McInnes et al., 2018) fitted on CLIP ViT-B/32 embeddings and fMRI encoder weights, both normalized unit-norm. To close this modality gap, we utilize a softmax weighted sum to project the voxel weights onto the space of natural images. Let the original voxel weight be wr e RRXM which we will assume to be unit-norm for convenience. We have a set with natural images M = {M1, MM,, M3, ... MM
In practice, natural image I' that achieves CLIP (I")
For each image, we compute the CLIP embedding e, CLIP ing (Mg). Given Wrnnig we use cosine similarity followed by softmax with temperature to compute a score that sums across all images. For each weight W.iigg and example image My:
exp( (W Worig /T)
Score
exp Ek k= W.neee eT/T)
We parameterize w. using a weighted sum derived from the scores, applied to the norms and directions of the image embeddings:
K
Σ Score,, llealle
Score & *
Ck Tex
In Figure 2(c) we show the cosine similarity between Worig and Wirooj as we increase the size of M. This projection operator can be treated as a special case of dot-product attention (Vaswani et al., 2017), with query = W.eTig key = (e,, c.. ..ck ,<x), ) and value equal to norm or direction of (ei, e2, ,ex). A similar approach is leveraged by Li al. (2023b), which shows similar operator outperforms nearest neighbor search for text-only caption inference. As W!T lies in the space of CLIP embeddings for natural images, this allows to leverage any existing captioning system that solely conditioned on the final CLIP embedding of an image. We utilize a frozen CLIPCap network, consisting of projection layer and finetuned GPT-2 (Mokady al., 2021).
4

---

Published as a conference paper at ICLR 2024
people
tennis
body
dress
Oiil
g3me
Faa Scissor
chair
food Caki
DeoplE
Wommnn
ba cackgruund
drees
Couple
Group
Fold
plute
Sireet
moonntain
Reld
background
sky
picture
wall
dla
celiphone
Fower
Sclsor
Woman
por
pointing air
group
3kV
cel phone
ower
clock
city
building
bridge
kitheen
Link
window
table
room
street
UMAP11
train
bed
racquet
tennis beseball
collphone
game
Souer
mountain
bridge
tle
build Bamstrr
kitchen
SPK
cale
UMAPP
Lable
bed
tran
Street
painting
buiding
window
city
man
clock
bed
porsen
mrtuure
Sink
womann peop
couple
bony
UMAP2
Cake
(b)
UMAP3
650
Superior
Superior
UMAP2
UMAP1
Anterior
Anterior
Right Hemisphere
(c)
Left Hemisphere
FRA
S1
(d)
$2
$5
S7
Figure 3: Interpreting the nouns generated by BrainSCUBA We take the projected encoder weights and fit a UMAP transform that goes to 4-dims. (a) The 50 most common noun embeddings across the brain are projected & transformed using the fMRI UMAP. (b) Flatmap of S1 with ROIs labeled. (c) Inflated view of $1. (d) Flatmaps of S2, S5, S7. We find that BrainSCUBA nouns are aligned to previously identified functional regions. Shown here are body regions (EBA), face regions [FFA-I/FFA- W/FFA-2/aTL-faco place regions (RSC/OPA/PPA), Note that the yellow near FFA match the food regions identified by Jain et al. (2023). The visualization style is inspired by Huth et al. (2016).
RESULTS
In this section, we utilize BrainSCUBA to generate voxel-wise captions and demonstrate that it can capture the selectivity in different semantic regions in the brain. We first show that the generated nouns are interpretable across the entire brain and exhibit a high degree of specificity within pre- identified Entegory-selectiie regions. Subsequently, diffusion models to generate novel images, and confirm the images are semantically consistent within their respective regions. Finally, utilize BrainSCUBA analyze the distribution of person representations across the brain to offer novel neuroscientific insight. These results illustrate BrainSCUBA's ability characterize human visual cortical populations, rendering it promising framework for exploratory neuroscience.
the captions input rext-to-image
4.1 SETUP
We utilize the Natural Scenes Dataset (NSD; Allen et al. (2022)), the largest whole-brain 7T human visual stimuli dataset. the 8 subjects, 4 subjects viewed the full 10, 000 image set repeated 3x. We use these subjects, S1, S2, S5, S7, for experiments in the main paper, and present additional results in the appendix. The fMRI activations (betas) are computed using GLMSingle (Prince et al., 2022), and further normalized each voxel's ''s response is 0, x² 1 session basis. The response across repeated viewings the same image is averaged. The brain encoder is trained on the 9000 unique images for each subject, while the remaining 1000 images viewed by all used to validate R?2
5

---

Published as a conference paper at ICLR 2024
Faces
Places
Bodies
Food
Words
S1
aagossase
$2
Figure 4: Top BrainSCUBA nouns via voxel-wise captioning in broad category selective regions. We perform vart-of-spechh tagging and lemmatization to extract the nouns, axis normalized by voxel count. We find that the generated captions are semantically related to the functional selectivity of broad category selective regions. Note that the word "close" tended to appear in the noun phrase "close-up", which explains its high frequency in the captions from food- and word-selective voxels.
The unpaired image projection set is 2 million combination of LAION-A v2 (6+ subset) and Open Images (Schuhmann al., 2022; Kuznetsova et al., 2020). We utilize OpenAl's ViT-B/32 for the encoder backbone and embedding computation this is the standard for CLIP conditioned caption generation, For image generation, we use the same model as used by Luo et al. (2023) in BrainDiVE, stable-di
with 50 steps of second order DPM-Solver++. In order to ensure direct comparability with BrainDiVE results, OpenCLIP's CoCa ViT-L/14 is used for image retrieval and pero-shot classification. We define face/place/body/woor regions using independent category localizer data provided with the NSD by Allen et al. (2022) (threshold of >> 2), and use the masks provided by Jain et al. (2023) define the food regions. For details the human study, please see the appendix.
4.2 VOXEL WISE TEXT GENERATIONS
In this section, we first investigate how BrainSCUBA outputs conceptually tile the higher visual cortex. We perform part-of-speech (POS) tagging and lemmatization of the BrainSCUBA output for four subjects, and extract the top-50 nouns. To extract noun specific CLIP embeddings, reconstitute them into sentences the form "A photo of a/an [NOUN]" as suggested by CLIP. Both the embeddings and the brain encoder voxel-wise weights are projected to the space of CLIP image embeddings and normalized to the unit-sphere for UMAP. We utilize UMAP fit on the encoder weights for S1. Results are shown in Figure 3. We observe that the nouns generated by BrainSCUBA are conceptually aligned to pre-identified functional regions. Namely, voxels in extrastriate body area (EBA) are selective nouns that indicate bodies and activities (green), fusiform face area (FFA-I/FFA- exhibits person/body noun selectivity (blue-green), place regions retrosplenial cortex (RSC), occipital place area (OPA), and parahippocampal place area (PPA) show selectivity for scene elements (magenta), and the food regions (yellow; Jain et al. (2023)) surrounding FFA exhibit selectivity for food-related nouns. These results show that our framework can characterize the broad semantic selectivity of visual cortex in zero-shot fashion.
We further quantify the top-10 nouns within each broad category selective region (Figure 4). We observe that BrainSCUBA generates nouns that are conceptually matched to the expected preferred category of each region. Note the multimodal selectivity for words/people/food within the word region has also been observed by Mei et al. (2010); Khosla & Wehbe (2022).
4.3 TEXT-GUIDED BRAIN IMAGE SYNTHESTS
Visualization of the captions can be helpful in highlighting subtle co-occurrence statistics, with novel images critical for future hypothesis driven investigations of the visual cortex (Ratan Murty et al., 2021; Gu et al., 2023; Jain et al., 2023). We utilize a text-to-image diffusion model, and condition the synthesis process on the voxel-wise captions within an ROI (Figure 5). We perform 1000 generations ROI, subsampling without replacement when the number of voxels/captions in a ROI exceed 1000, and randomly sample the gap when there are fewer than 1000. For face-, place-, word-, body-selective regions, visualize the top-5 out of 10, 000 images ranked by real

---

Published as a conference paper at ICLR 2024
Place Voxels
Word Voxels
A50
STOP
STOP
Brainivs
PATIUS WANKSNS
8016
Brinscuu
Face Voxels
Body Voxels
Bolndie
BIANSCUSS
Food Voxels
BOlROVE
BrANSCUUA
Figure 5: Novel images for category fMRI stimuli and generated images for the place/word/facc/body regions, and the top- 10 images for the food region. We observe that images generated with BrainSCUBA appear more coherent.
selective voxels in S2. We visualize the top-5 images from the
Faces
Places
Bodies
Food
Words
NSO all
top-1%
BrainDIvE
BrainSCUBA
S2
30
NSD
top-1%
BrainDiVE
BrainSCUBA
S5
Figure Evaluating the distribution of BrainSCUBA captions with different encoder. We train an encoder with different backbone (EVA02-CLIP-B- 16) from both BrainDiVE and BrainSCUBA. For each region, we evaluate the response to all images a subject saw in NSD, the response of the top-1% of images in NSD stimuli ranked using EVA02, the top-10% of images generated by BrainDiVE and BrainSCUBA and ranked by their respective encoders. Each region is normalized to achieve high predicted responses despite not performing explicit gradient based maximization like BrainDiVE, and yields concretely interpretable captions. BrainSCUBA is also 10x faster per image.
[-1,1] 1] using the min/max of the predicted responses NSD stimuli. BrainSCUBA
average ROI response from the fMRI stimuli (NSD), and the top-5 out of 000 generations ranked by predicted response using the respective BrainDiVE (Luo et al., 2023) and BrainSCUBA encoders. BrainDiVE used for comparison as it is the state of the art method for synthesizing activating images in the higher visual cortex, and we follow their evaluation procedure. For the food region, visualize the 10p Predicted activation is shown Figure 6, with semantic classification shown in Table 1. Visual inspection suggests method can generate diverse images semantically aligned
7

---

Published as a conference paper at ICLR 2024
Faces
Places
Bodies
Words
Food
Mean
S2
S5
S2
S5
S2
S5
S2
S5
S2
S5
S2
S5
NSD all stim
17.1
17.5
29.4
30.7
31.5
30.3
11.0
10.1
10.9
11.4
20.0
20.0
NSD top-100
45.0
43.0
78.0
93.0
59.0
55.0
48.0
33.0
86.0
83.0
63.2
61.4
Br 100
68.0
64.0
100
100
69.0
77.0
61.0
80.0
94.0
87.0
78.4
81.6
BrainSCUBA-100
67.0
62.0
100
99.0
54.0
73.0
55.0
34.0
97.0
92.0
74.6
72.0
Table 1: Semantic evaluation of images with zero-shot CLIP. We use CLIP to perform zero-shot 5-way classification. Show here is the percentage where category the image matches the preferred category for a brain region. This is shown for each subject's NSD stimulus set (10, 000 images for S2&55); the top- 100 images (top-1 9) evaluated by average region true fMRI, the top- 100 (10%) of BrainDiVE and BrainSCUBA (bolded) evaluated by their respective encoders. BrainSCUBA has selectivity that is closer the true NSD top 1%.
with the target category. Our images are generally more visually coherent than those generated by BrainDiVE, and contain more clear text in word voxels, and fewer degraded faces and bodies in the respective regions. This is likely because our images are conditioned on text, while BiainDiVE utilizes the gradient signal alone.
4.4 INVESTIGATING THE BRAIN'S SOCIAL NETWORK
The intrinsic social nature of humans significantly influences visual perception. This interplay is evident in the heightened visual sensitivity towards social entities such as faces and bodies (Pitcher & Ungerleider, 2021; Kanwisher et al., 1997; Downing et al., 2001). In this section, we explore BrainSCUBA can provide insights the finer-grad linr-grainedd coding of people in the brain. We use rule based filter and count the number of captions that contain one of 140 nouns that describe people (person, man, woman, child, boy, girl, family, occupations, and plurals). We visualize the voxels whose captions contain people in Figure 7, and provide a quantitative evaluation in Table 2. We observe that our captions can correctly identify non-person-sede selective scene, food, and word regions as having lower person content than person-selective ROIs like the FFA or the EBA. Going beyond traditional functional ROIs, we find that the precuneus visual area (PCV) and the temporoparietal junction (TPJ) have a very high density of captions with people. The precuneus has been implicated in
mental representations of self (Cavanna & Trimble, 2006; Petrini al., 2014), while the TPJ has been suggested to be involved in theory of mind and social cognition (Saxe & Kanwisher, 2013). Our results lend support to these hypotheses.
(a)
(b)
S1
FEA-1
facess
57
FFA-2
0.0
1.0
Person existence density
Figure 7: Presence of people in captions. We perform rule-based filtering and identify voxels where a caption contains at least one person. Data is surface smoothed for visualization. Dotted orange oval shows approximate location of TPJ, which is linked to theory of mind; green circle shows location of PCV, associated with third-person perspective of social interactions. Note that TPJ is HCP defined following Igelström & Graziano (2017), while PCV is HCP atlas region 27 (Glasser et al., 2016).
A close visual examination of Figure 7 suggests a divide within EBA. We perform spherical k- clustering on joint encoder weights for > 2 EBA from S1/S2/S5/S7, and identify two stable clusters. These clusters are visualized in Figure 8. Utilizing the rule parser, we labels the voxels into those that contain a single individual or multiple people, and further visualize the top-nouns within each of these two clusters. While both clusters include general person words like "man" and "woman". cluster I has more nouns that suggest groups of people interacting together (group, game, people), and cluster 2 has words that suggest close-ups of individuals with objects that may be hand-held. To validate our findings, we perform a study where subjects are asked to evaluate the top-100 100 images from each of the clusters. Results shown in Table 3. Aligned with the top-nouns, the study suggests that
8

---

Published as a conference paper at ICLR 2024
Non-Person
Person
Other
RSC
OPA
PPA
Food
Word
EBA
FFA
PCV
TPJ
S1
12.9
17.3
10.6
11.5
32.0
87.2
88.5
89.7
92.1
S2
5.58
8.15
2.70
20.0
34.8
81.4
87.2
70.8
89.1
S5
9.31
6.43
1.95
17.8
38.4
79.5
89.4
78.5
79.9
S7
7.14
9.87
5.99
10.7
36.9
84.3
89.5
84.2
90.3
Mean
8.72
10.4
5.30
15.0
35.5
83.1
88.6
80.8
87.8
Table 2: Percentage of captions in each region that contain people. We observe a sharp difference between non-person regions (Scene RSC/OPA/PPA, Food, Word), and regions that are believed to be person selective (body EBA, face FFA). We also observe extremely high person density in PCV - region involved in third- person social interactions, and TPJ region involved in social self-other distinction.
(a)
(b)
1.0
S1
S5
S5
Multi
Cluster
Cluster
0.0
1.0
Single
(c)
S1
$2
S5
S7
Cluster
Cluster
of 4088
& Lafes
Figure 8: Clusters within EBA. (a) The EBA clusters for two subjects are shown on flatmap. (b) Number of people mentioned in each caption. (c) Top nouns within each cluster, y-axis is normalized to the number of voxels within a cluster. Compared to Cluster-1, Cluster-2 has less emphasis on multiple people and more emphasis objects that can be held.
Which cluster is more...
people per-ing
inanimate objs
far away
sports
S1
$2
S5
S7
S1
$2
S5
S7
S1
S2
S5
S7
S1
S2
S5
S7
EBA-I (Cluster 1) EBA-2 (Cluster 2)
88
84
11
12
78
63
5
10
13
72
80
81
65
21
21
14
25
9
12
11
Table 3: Human evaluation comparing two EBA clusters.. Evaluators compare the top 100 images for each cluster, with questions like "Which group of images is more X?", answers include EBA- I/EBA-2/Same We do not show "Same": responses sum to 100 across all three options. Results in %.
cluster-1 has more groups of people, fewer inanimate objects, and consists of larger scenes. This intriguing novel finding about the fine-grained distinctions EBA can lead to new hypotheses about its function. This finding also demonstrates the ability of BrainSCUBA to uncover broad functional differences across the visual cortex.
DISCUSSION
Limitations and Future Work. Although our methods can generate semantically faithful de- scriptions for the broad category selective regions, our approach ultimately relies on pre-trained captioning model. Due to this, our method reflects the biases of the captioning model. It further not clear if the most selective object in each region can be perfectly captured by language. Future work could explore the use of more unconstrained captioning models (Tewel et al., 2022) or more powerful language models (Touvron et al., 2023).
To summarize, in this paper we propose BrainSCUBA, method which can generate voxel-wise captions to describe each voxel's semantic selectivity. We explore how the output tiles the higher visual cortex, perform text-conditioned image synthesis with the captions, and apply it to uncover finer-grained patterns of selectivity in the brain within the person class. Our results suggest that BrainSCUBA may used to facilitate data-driven exploration of the visual cortex.
Conclusion.

---

Published as a conference paper at ICLR 2024
6 ETHICS
Institutional review board approval was requested and granted for the human study on body region images. The NSD dataset contains 8 subjects from age 19 32; half white, half asian; men, women. These demographics are not necessarily reflective of the broader population. As part of our framework, we use a pretrained language model based on fine-tuned GPT-2, and will reflect the biases present in the training and fine-tuning data. We believe language models trained on more diverse datasets, or those trained with weaker constraints will help mitigate issues of bias. We also note that text-conditioned image diffusion models reflect negative stereotypes, but such issues could be mitigated on a case-by-case basis using concept editing (Gandikota et al., 2024).
7
ACKNOWLEDGMENTS
This work used Bridges-2 Pittsburgh Supercomputing Center through allocation SOC220017 from the Advanced Cyberinfrastructure Coordination Ecosystem: Services & Support (ACCESS) program, which is supported by National Science Foundation grants #2138259, #2138286, #2138307, #2137603, and #2138296. We also thank the Carnegie Mellon University Neuroscience Institute for support.
REFERENCES
Emily J Allen, Ghislain Yves,,, Yihan Wu, Jesse L Breedlove, Jacob S Prince, Logan T Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, et al. A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence. Nature nearoxcience,
2022. 1, 5,
Pouya Bashivan, Kohitij Kar, and James DiCarlo. Neural population control via deep image synthesis. Science, 364(6439):cav94330 2019. 3
Judy Borowski, Roland S Zimmermann, Judith Schepers, Robert Geirhos, Thomas SA Wallis, Matthias Bethge, and Wieland Brendel. Exemplary natural images explain enn activations better than ofate-of-the-art feature visualization. arXiv preprint arXiv:2010. /2606, 2020.
Andrea E Cavanna and Michael R Trimble. The precuneus: review of its functional anatomy and behavioural correlates. Brain, 129(3):564-583, 2006.
Zijiao Chen, Jiaxin Qing, Tiange Xiang, Wan Lin Yue, and Juan Helen Zhou. Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding. arXiv preprint arXiv:2211.cmm 069556, 1(2):4, 2022. 2
Zijiao Chen, Jiaxin Qing, Tiange Xiang, Wan Lin Yue, and Juan Helen Zhou. Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patten Recognition, pp. 22710-22722 2023.
Mehdi Cherti, Romain Beaumont, Ross Wightman, Mitchell Wortsman, Gabriel Ilharco, Cade Gordon, Christoph Schuhmann, Ludwig Schmidt, and Jenia Jitsev. Reproducible scaling laws for contrastive Language-image learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2818 11899, 2023.
Laurent Cohen, Stanislas Dehaene, Lionel Naccache, Stéphane Lehéricy, Ghislaine Dehaene- Lambertz, Marie-Anne Hénaff, and François Michel. The visual word form area: spatial and temporal characterization of an initial stage of reading in normal subjects and posterior split-brain patients. Brain, 123(2):291-307. 2000. 2
Colin Conwell, Jacob S. Prince, Kendrick N. Kay, George A. Alvarez, and Talia Konkle. What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation brains and machines? bioRxiv, 2023. doi: 10.1101/2022.00 03. 48.88888.. 2,
10

---

Published as a conference paper at ICLR 2024
Fatma Deniz, Anwar O Nunez-Elizalde Alexander G Huth, and Jack L Gallant. The representation of semantic information across human cerebral cortex during listening versus reading is invariant to stimulus modality. Journal of Neuroscience, 39(39):7722-7735 2019. 2
Robert Desimone, Thomas D Albright, Charles Gross, and Charles Bruce. Stimulus selective properties of inferior temporal neurons in the macaque. Journal of Neuroscience, 4(8):2051-206 1984.
Adrien Doerig, Tim C Kietzmann, Emily Allen, Yihan Wu, Thomas Naselaris, Kendrick Kay, and lan Charest. Semantic scene descriptions as an objective of human vision. arXiv preprint arXiv:2209. 1737, 2022. 3,3
Paul E Downing, Yuhong Jiang, Miles Shuman, and Nancy Kanwisher. A cortical area selective for visual processing of the human body. Science, 293(5539):2400-4477 2001. 2,
Michael Fickenberg, Alexandre Gramfort, Gaäl Varoquaux, and Bertrand Thirion. Seeing it all: Convolutional network layers map the function of the human visual system. Neurolmage, 152: 184-194, 2017.
Russell Epstein and Nancy Kanwisher. cortical representation of the local visual environment. Nature, 392(6676):598-601, 1998. 1,2
Gidon Felsen and Yang Dan. A natural approach studying vision. Nature neuroscience, 8(12): 1643-1646, 2005. 2
Matteo Ferrante, Furkan Ozcelik, Tommaso Boccato, Rufin VanRullen, and Nicola Toschi. Brain captioning: Decoding human brain activity into images and text, arXiv preprint arXiv:2305.11560, 2023. 2,
Jack L Gallant, Charles E Connor, and David C Van Essen. Neural activity in areas vl, v2 and v4 during free viewing of natural scenes compared to controlled viewing. Neuroreport, 9(7):
1678, 1998. 2
Rohit Gandikota, Hadas Orgad, Yonatan Belinkov, Joanna Materzyúska, and David Bau. Unified concept editing in diffusion models. /EEE/CVF Winter Conference on Applications of Computer Vision, 2024. 10
Isabel Gauthier, Marlene Behrmann, and Michael J Tarr. Can face recognition really be dissociated from object recognition? Journal of cognitive neuroscience, (4))34993370 1999.
Matthew F Glasser, Timothy S Coalson, Emma C Robinson, Carl D Hacker, John Harwell, Essa Yacoub, Kamil Ugurbil, Jesper Andersson, Christian F Beckmann, Mark Jenkinson, et al. A multi-modal parcellation of human cerebral cortex. Nature, 536(7615): 2016. 8
Kalanit Grill-Spector. The neural basis of object perception. Current opinion in neurobiology, 13(2): 159-166, 2003. 2,2
Zijin Gu, Keith Wakefield Jamison, Meenakshi Khosla, Emily Allen, Yihan Wu, Ghislain St- Yves, Thomas Naselaris, Kendrick Kay, Mert R Sabuncu, and Amy Kuceyeski. NeuroGen: activation optimized image synthesis for discovery neuroscience. Neurolmage, 247: 18812, 2022. 2,
Zijin Gu, Keith Jamison, Mert R Sabuncu, and Amy Kuceyeski. Human brain responses modulated when exposed to optimized natural images or synthetically generated images. Communications Biology, 6(1):1076, 2023. 6
Kuan Han, Haiguang Wen, Junxing Shi, Kun- Kun-Han Lu, Yizhen Zhang, Di Fu, and Zhongming Liu. Variational autoencoder: An unsupervised model for encoding and decoding fmri activity in visual cortex. Neurolmage, 198: 126-136, 2019. 2
Alexander G Huth, Shinji Nishimoto, An T Vu, and Jack L Gallant. A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron, 76(6):2200-122: 2012. 2
11

---

Published as a conference paper at ICLR 2024
Alexander G Huth, Wendy A De Heer, Thomas L Griffiths, Frédéric E Theunissen, and Jack L Gallant. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature, 532 (7600):453-44) -4404558,,8,,,, 2016. 2,5
Kajsa M Igelström and Michael SA Graziano. The inferior parietal lobule and temporoparietal junction: network perspective. Neuropsychologia, 105:70- 83, 2017. 8
Nidhi Jain, Aria Wang, Margaret M. Henderson, Ruogu Lin, Jacob S. Prince, Michael J. Tarr, and Leila Wehbe. Selectivity for food in human ventral visual cortex. Communications Biology 2023 doi: 10. 10338442005-0022-4455662 5,
6:1, 6:1-14,14 2023. ISSN 2399-3642.
Yukiyasu Kamitani and Frank Tong. Decoding the visual and subjective contents of the human brain. Nature neuroscience, 8(5):679-68,, 2005. 2
Nancy Kanwisher, Josh McDermott, and Marvin M Chun. The fusiform face area: module in human extrastriate cortex specialized for face perception. Journal of neuroscience, 17(1 17(11)4302-4311, 1997. 2,
Meenakshi Khosla and Leila Wehbe. High-level visual areas act like domain-general filters with strong selectivity and functional specialization. bioRxiv, pp. 2022-03, 2022.
Meenakshi Khosla, N. Apurva Ratan Murty, and Nancy Kanwisher. highly selective response to food in human visual cortex revealed by hypothesis-free voxel decomposition. Current Biology, 32:1-13, 2022. 1,2
Jonas Kubilius, Martin Schrimpf, Kohitij Kar, Rishi Rajalingham, Ha Hong, Najb Majaj. Elias Issa, Pouya Bashivan, Jonathan Prescott- Roy, Kailyn Schmidt, et al. Brain-like object recognition with nigh-performing shallow recurrent anns. Advances in neural information processing systems, 32, 2019. 2
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont- Font-Tuset, Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, et al. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. International Journal of Computer Vision, 128(7):1956-1981, 2020. 6
Junnan Li, Dongxu Li, Silvio Savarese, and Steven Hoi. Blip-2: Bootstrapping language- sunuage--maae pre- training with frozen image encoders and large language models. arXiv preprint arXiv:2301.12597, 2023a.
Wei Li, Linchao Zhu, Longyin Wen, and Yi Yang. Decap: Decoding clip latents for zero-shot captioning via text-only training. arXiv preprint arXiv:2303.0303 2023b. 2,
Yulong Liu, Yongqiang Ma, Wei Zhou, Guibo Zhu, and Nanning Zheng. Brainclip: Bridging brain and visual-linguistic representation via clip for generic natural visual stimulus decoding from fmri arXiv preprint ar axXiv:2302.1297. 2023. 2,
Yizhuo Lu, Changde Du, Diampeng Wang, and Huiguang He. Minddiffuser: Controlled image reconstruction from human brain activity with semantic and structural diffusion. arXiv preprint arXiv:2303.14139, 2023. 2
Andrew Luo, Margaret M Henderson, Leila Wehbe, and Michael J Tar. Brain diffusion for visual ex- ploration: Cortical discovery using large seale generative models. arXiy preprint arXiv:2306.0300 2023. 2, 6,
Eleanor Maguire. The retrosplenial contribution to human navigation: a review of lesion and
neuroimaging findings. Scandinavian journal of psychology, 42(3):225-238, 2001. 1,
Weijian Mai and Zhijun Zhang. Unibrain: Unify image reconstruction and captioning all in one diffusion model from human brain activity. arXiv preprint arXi::2308.07423 2023.
Gregory McCarthy, Aina Puce, John C Gore, and Truett Allison. human fusiform gyrus. Journal of cognitive neuroscience, 95):600--610, 1997. 1
Face-specific processing in the
12

---

Published as a conference paper at ICLR 2024
Leland McInnes, John Healy, and James Melville. Umap:
Uniform manifold approximation and projection for dimension reduction. arXiv preprint 18X.::800..3326, 2018. 4
Leilei Mei, Gui Xue, Chuansheng Chen, Feng Xue, Mingxia Zhang, and Qi Dong. The "visual word form area" is involved in successful memory encoding of both words and faces. Neuroimage, 52 (1):371-378, 2010.
Ron Mokady, Amir Hertz, and Amit Bermano. Clipeap: Clip prefix for image captioning. arXiv preprint (tt::2111.097344 2021. 2, 4
Thomas Naselaris, Kendrick N Kay, Shinji Nishimoto, and Jack Gallant. Encoding and decoding in fmri. Neuroimage, 56(2):400-410, 2011. 2
Furkan Ozcelik and Rufin VanRullen. Brain- diffuusrfff Natural scene reconstruction from fmri signals using generative latent diffusion. arXiv preprint arXiv:2303.0533. 2023.
Ian ML Pennock, Chris Racey, Emily Allen, Yihan Wu, Thomas Naselaris, Kendrick Kay, Anna Franklin, and Jenny M Bosten. Color-biased regions in the ventral visual pathway are food selective, Current Biology, 33(1):134-146, 2023. 1, 2
Karin Petrini, Lukasz Piwek, Frances Crabbe, Frank E Pollick, and Simon Garrod. Look at those two!: The precuneus role in unattended third-person perspective of social interactions. Human Brain Mapping, 35(10):51 (11(0)))519-52000 2014.
David Pitcher and Leslie G. Ungerleider. Evidence for a third visual pathway specialized for social perception. Trends in Cognitive Sciences, 25:100-110, 2021. ISSN 1364-6613. doi:
(..1016J.TTCS.2022 11.06.
Carlos R Ponce, Will Xiao, Peter F Schade, Till S Hartmann, Gabriel Kreiman, and Margaret Livingstone. Evolving images for visual neurons using a deep generative network reveals coding principles and neuronal preferences. Cell, 177(4):999-009, 2019.
Sara F Popham, Alexander G Huth, Natalia Y Bilenko, Fatma Deniz. James Gao, Anwar O Nunez- Elizalde, and Jack L Gallant. Visual and linguistic semantic representations are aligned at the border of human visual cortex. Nature neuroscience, 24(1 111211666--166, 2021. 2
Jacob S Prince, Ian Charest, Jan W Kurzawski, John A Pyles, Michael J Tarr, and Kendrick N Kay. Improving the accuracy of single-trial fmri response estimates using glmsingle. eLife, 11:777599, nov 2022. ISSN 2050-084X. doi: (0.7554/el.ife.775599
Aina Puce, Truett Allison, Maryam Asgari, John C Gore, and Gregory McCarthy. Differential sensitivity of human visual cortex to faces, letterstrings, and textures: a functional magnetic resonance imaging study. Journal of neuroscience, 16(16):5205-77) 5215,, 1996. 1.2
Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021. 2,
Apurva Ratan Murty, Pouya Bashivan, Alex Abate, James DiCarlo, and Nancy Kanwisher, Com- putational models of category-selective brain regions enable tigh-throughhut tests of selectivity. Nature communications, 12(1):5540, 2021. 2. 3,
Ziqi Ren, Jie Li, Xuetong Xue, Xin Li, Fan Yang, Zhicheng Jiao, and Xinbo Gao. Reconstructing seen image from brain activity by visually-guided cognitive representation and adversarial learning. Neurolmage, 228:117602. 17602, 2021.
Rebecca Saxe and Nancy Kanwisher. People thinking about thinking people: the role of the temporo- parietal junction in "theory of mind". In Social neuroscience, pp. 171-182. Psychology Press, 2013. 8
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. arXiv preprint 20.::22100004444 2022. 6
13

---

Published as a conference paper at ICLR 2024
Katja Seeliger, Umut Güşlü, Luca Ambrogioni, Yagmur Güęlüütürk, and Marcel AJ van Gerven. Generative adversarial networks for reconstructing natural images from brain activity. Neurolmage. 181:775-785, 2018. 2
Guohua Shen, Tomoyasu Horikawa, Kei Majima, and Yukiyasu Kamitani. Deep image reconstruction from human brain activity. PLoS computational biology, 15(T)=e1006633 2019. 2
Sheng Shen, Liunian Harold Li, Hao Tan, Mohit Bansal, Anna Rohrbach, Kai-Wei Chang, Zhewei Yao, and Kurt Keutzer. How much can clip benefit assion-and-languag tasks? arXiv preprint arXiv:2107.06383, 2021. 2
Quan Sun, Yuxin Fang, Ledell Wu, Xinlong Wang. and Yue Cao. Eva-clip: Improved training techniques for clip at scale. arXiv preprint 2rXiv:2303.15389 2023.
Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. bioRxiv, pp. 2022- 11, 2022.
Yoad Tewel, Yoav Shalev, Idan Schwartz, and Lior Wolf. Zerocap: Zero-shot image-to-text generation for visual-semantic arithmetic. In Proceedings of the IEEF/CVF Conference on Computer Vision and Puttern Recognition, pp. 77918-17928 2022. 2,9
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.092886 2023. 9
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural information processing systems, 30, 2017. 4
Edgar Y Walker, Fabian H Sinz, Erick Cobos, Taliah Muhammad, Emmanouil Froudarakis, Paul G Fahey, Alexander Ecker, Jacob Reimer, Xaq Pitkow, and Andreas Tolias. Inception loops discover what excites neurons most using deep predictive models. Nature neuroscience, 22(12): 2060-2065, 2019. 3
Aria Yuan Wang, Kendrick Kay, Thomas Naselaris, Michael J Tarr, and Leila Wehbe. Incorporating natural language into vision models improves prediction and understanding of higher visual cortex. BioRxiv, pp. 2022-09, 2022. 2, 3
Haiguang Wen, Junxing Shi, Wei Chen, and Zhongming Liu. Deep residual network predicts cortical representation and organization of visual features for rapid categorization. Scientific reports, 8(1); 3752, 2018.2 2
Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James J DiCarlo. Performance-optimizze hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, 111(23)8819-8624, 2014. 2
Roland S Zimmermann, Judy Borowski, Robert Geirhos, Matthias Bethge, Thomas Wallis, and Wieland Brendel. How well do feature visualizations support causal understanding of cnn activa- tions? Advances in Neural Information Processing Systems, 34:11730-1 11744, 2021.
14

---

Published as a conference paper at ICLR 2024
A
APPENDIX
Sections
Visualization of each subject's top-nouns for category selective voxels (A.1)
2. Visualization of UMAPs for all subjects (A.2)
3. Novel image generation for all subjects (A.3)
4. Distribution of "person" representations across the brain for all subjects (A.4)
5. Additional extrastriate body area (EBA) clustering results (A.5)
6. Human study details (A.6)
7. Training and inference details (A.7)
8. Top adjectives and more sentences (A.8)
9. Encoder fitting stability (A.9)
10. Ground truth functional localizer category distribution (A.10)
11. Fine-grained concept distribution outside EBA (A.11)
12. Norm of the embeddings with and without decoupled projection (A.12)
15

---

Published as a conference paper at ICLR 2024
A.1
VISUALIZATION EACH SUBJECT'S TOP-NOUNS FOR CATEGORY SELECTIVE VOXELS
Faces
Places
Bodies
Food
Words
51
spp fff
52
18
20 tppede / papperse war
24-
244
pepees apppe puppe0s0
18
14-
18
16.
57
24
11.
58
Figure S.1: Top BrainSCUBA nouns via voxel-wise captioning broad category selective regions for all subjects. We see broad semantic alignment between the top-nouns and the semantic selectivity of region. Note that the category selective voxels were derived from the intersection of official NSD functional localizer values and their provided region masks.
16

---

Published as a conference paper at ICLR 2024
A.2 VISUALIZATION UMAPS FOR ALL SUBJECTS
heir
body
aress
SCTSSO
man SrOUD
Cousle
mountein
fiver sky
Sack weckgrund
SC9S01
Tenms
FRCQUIE
coupple
group
game
chay
tackground
GPCSS
Held
Kow
kitchen
bridge
Tower
mall
basetall
glats
cellohane
Rower
betehhll
person
UMAP2
background
mounssin
pct.ro
sky 1d
dOck
Jotchen
window
body
bridge
glans
VINCE
SIX
CWCE
Bed
Tram
parniing
Street
kinceen
Window
city
clock
gemee
g11
food
Eody
pote
FIVer
Dec
UMAP3
UMAP2
UMAPL
OBR
ESA
OTL
$1
FRA
S2
S3
Superion
Suporior
Anterior
Anterior
Right Hemisphere
Left Hemisphere
Figure S.2: UMAP transform results for S1-S4. All vectors are normalized to unit norm prior to UMAP. UMAP is fit on S1. Both word and voxel vectors projected onto the space of natural images prior to transform using softmax weighted sum. Nouns are the most common across all subjects.
17

---

Published as a conference paper at ICLR 2024
food
VORTAAN
sky
lair
people
per
FSCQUET
Sult
bosy
couple
group
Cress
hand
gamm
UNMPP1
Laba
pesple
drees
Cousle Paintin
OHf air
kitchen
plato
UMAPP1
food
Tald
bridge
Power
background
person
gias
worenn
celphone
UMAP2
Cake
nd1d
Clock
9gh
buising
kotchen
coble
body
picture
bringe
build Himmtuue
train
Window
city
clock
chair
food
bo0y
background
train
PROquet
Devebal
EpPione
glaas
kitchen
(000
muuuur
gamma
UMAP3
UMAP1
OBA
A
2
S6
Superior
Right Hemisphere
Left Hemisphere
S8
Figure S.3: UMAP transform results for S5-S8. All vectors normalized to unit norm prior to UMAP. UMAP is fit on S1. Both word and voxel vectors are projected onto the space of natural images prior transform using softmax weighted sum. Nouns are the most common across all subjects.
18

---

Published as a conference paper at ICLR 2024
A.3 NOVEL IMAGE GENERATION FOR ALL SUBJECTS
Place Voxels
Word Voxels
NSD
TAOS
BroinDiiv (Gradient)
a
WOD 2116
BrainsCUUA (Curs))
Face Voxels
Body Voxels
NSD
BrainDive (Gradientt
BrainSCUBA (ours)
Food Voxels
NSD (MMRI)
BrainDiiv (Gradient)
ISauASCUBA (Chirs)
Figure S.4: Image generation for S1. We visualize the top-5 for face/place/body/ver categories, and the top-10 for food. NSD images are ranked by ground truth response. BrainDiVE and BrainSCUBA are ranked by their respective encoders. BrainSCUBA images have more recognizable objects and fewer artifacts, likely due to the use of captions rather than gradients as in BrainDiVE.
19

---

Published as a conference paper at ICLR 2024
Place Voxels
Word Voxels
ASD
STOP
STOP
BrainDiiv (Gradiem))
FATTUS
BrainnSC (Curs)
Face Voxels
Body Voxels
NSD (HOM8)
Brainpive (Gradient)
Brainscuua UBA (Ouns)
Food Voxels
NSD
BrainDive (Gradienm)
BralinSc/BA (Curs)
Figure S.5: Image generation for S2. We visualize the top-5 for face/place/body//oord categories, and the top-10 for food. NSD images are ranked by ground truth response. BrainDiVE and BrainSCUBA are ranked by their respective encoders. BrainSCUBA images have more recognizable objects and fewer artifacts, likely due the of captions rather than gradients in BrainDiVE.
20

---

Published as a conference paper at ICLR 2024
Place Voxels
Word Voxels
NEMEGH STOP OPPN
NSD
Brainpive (Gradient)
RATARY
RICKT
BrainSCuBA (Ourg)
R
Face Voxels
Body Voxels
NSD
Brainpive (Gradient)
BrainSCUBA BA (Duns)
Food Voxels
NSD
Boinnbive (Gradient)
BrainScuBA (Ours)
Figure S.6: Image generation for $3. We visualize the top-5 for face/place/body/worr categories, and the top-10 for food. NSD images are ranked by ground truth response. BrainDiVE and BrainSCUBA are ranked by their respective encoders. BrainSCUBA images have more recognizable objects and fewer artifacts, likely due to the use of captions rather than gradients as in BrainDiVE.
21

---

Published as a conference paper at ICLR 2024
Place Voxels
Word Voxels
NS
STOP
Brainpive (Gradient)
SEBCLUST
Wit fndet
4B
BrainScuBa (Ourg)
Face Voxels
Body Voxels
NSD
Brainpivv (Gradient)
BrainSCUBA (Ouns)
Food Voxels
NSD
BaainDive (Curs) (Sraclent)
ALl
BrainScUBA
Figure S.7: Image generation for S4. We visualize the top-5 for face/place/body/woor categories, and the top-10 for food. NSD images are ranked by ground truth response. BrainDiVE and BrainSCUBA are ranked by their respective encoders. BrainSCUBA images have more recognizable objects and fewer artifacts, likely due to the use of captions rather than gradients as in BrainDiVE.
22

---

Published as a conference paper at ICLR 2024
Place Voxels
Word Voxels
NSO (1MRI))
BrainDiiv (Gradiont)
FLovner
Swo
CAJU
LEET
Las Eargy Kan
BraiinSc (Curs)
Face Voxels
Body Voxels
NSD
BrainDive (Gradlent)
Brainscu BA (Ouns)
Food Voxels
NSD (1MRY)
BrainDive (Curs) (Gradient)
BralnnSc/BA
Figure S.8: Image generation for S5. We visualize the top-5 for face/place/body/wooo categories, and the top-10 for food. NSD images are ranked by ground truth response. BrainDiVE and BrainSCUBA are ranked by their respective encoders. BrainSCUBA images have more recognizable objects and fewer artifacts, likely due the of captions rather than gradients in BrainDiVE.
23

---

Published as a conference paper at ICLR 2024
Place Voxels
Word Voxels
BrainDWie (Gradiont)
bucto
BrainnSC (CurS)
& PAR
ANDDERS
Face Voxels
Body Voxels
NSD
Brainpive (Gradient)
Brainscu BA (Ouns)
Food Voxels
NSD
BralinScUBA BrainDive (CurS) (Gradien))
EL
Figure S.9: Image generation for S6. We visualize the top-5 for face/place/bodd//woo categories, and the top-10 for food. NSD images are ranked by ground truth response. BrainDiVE and BrainSCUBA are ranked by their respective encoders. BrainSCUBA images have more recognizable objects and fewer artifacts, likely due the of captions rather than gradients in BrainDiVE.
24

---

Published as a conference paper at ICLR 2024
Place Voxels
Word Voxels
STOP
ASD
(HMRI)
BrainDliv (Gradien))
BrainSC (Curs)
Face Voxels
Body Voxels
NSD
Brainpive (Gradient)
Brain Brainscy BBA (Ouns)
Food Voxels
NSD
BrainDive (Gradien))
Bra BralinScUABA (Curs)
Figure S.10: Image generation for S7. We visualize the top-5 for face/place/body/worr categories, and the top-10 for food. NSD images are ranked by ground truth response. BrainDiVE and Brain- SCUBA are ranked by their respective encoders. BrainSCUBA images have more recognizable objects and fewer artifacts, likely due the use of captions rather than gradients as BrainDiVE.
25

---

Published as a conference paper at ICLR 2024
Place Voxels
Word Voxels
ASD
STOP
BrainDliv (Gradiom))
ALISA
CAE
ING
BrainnSc (Curs)
SU SSAE
Face Voxels
Body Voxels
NSD
Brainpive (Gradient)
BrainscuBa BA (Oun))
Food Voxels
NSD (AMRY
BrainDive (Gradient)
Solne CANDAN
BralinScUBA
Figure S.11: Image generation for S8. We visualize the top-5 for face/place/body/word categories, and the top-10 for food. NSD images are ranked by ground truth response. BrainDiVE and Brain- SCUBA are ranked by their respective encoders. BrainSCUBA images have more recognizable objects and fewer artifacts, likely due the use of captions rather than gradients as BrainDiVE.
26

---

Published as a conference paper at ICLR 2024
A.4
DISTRIBUTION OF "PERSON" REPRESENTATIONS ACROSS THE BRAIN FOR ALL SUBJECTS
(a)
(b)
S1
OP4
bia EBA
FFA-1
PPN
aTL-faces
S2
S4
mTL
miL- bodies
bodies
S6
S7
S8
0.0
1.0
Person existence density
Figure 12: Presence of people in captions for S1-S8. (a) Flatmap of cortex. (b) Inflated map of cortex. Dotted orange oval shows approximate location of TPJ, which is linked to theory of mind: green circle shows location of PCV, associated with third-person perspective of social interactions. For S5 alone we additionally label the mTL-bodies area.
27

---

Published as a conference paper at ICLR 2024
Non-Person
Person
Other
RSC
OPA
PPA
Food
Word
EBA
FFA
PCV
TPJ
S1
12.9
17.3
10.6
11.5
32.0
87.2
88.5
89.7
92.1
S2
5.58
8.15
2.70
20.0
34.8
81.4
87.2
70.8
89.1
S3
6.57
16.9
4.49
24.4
33.9
84.7
90.3
75.3
83.2
S4
4.40
14.7
4.47
20.0
37.8
78.9
90.3
66.5
88.9
S5
9.31
6.43
1.95
17.8
38.4
79.5
89.4
78.5
79.9
S6
16.7
28.2
6.93
27.1
48.8
91.8
97.8
75,4
79.1
S7
7.14
9.87
5.99
10.7
36.9
84.3
89.5
84.2
90.3
S8
15.7
30.9
9.84
42.6
57.5
86.7
96.2
71,2
89.7
Mean
9.78
16.5
5.86
21.8
40.0
84.3
91.2
76.4
86.5
Table S.1: Percentage of captions in each region that contain people for S1-S8. We observe sharp difference between non-peree non-person regions (Scene RSC/OPA/PPA, Food. Word), and regions that are believed be person selective (body EBA, face FFA). We also observe extremely high person density in PCV region involved in third-person social interactions, and TPJ - a region involved in social self-other distinction.
28

---

Published as a conference paper at ICLR 2024
A.5
ADDITIONAL EXTRASTRIATE BODY AREA (EBA) CLUSTERING RESULTS
(a)
S1
S2
S5
S7
Cluster
Cluster
(b)
S1
S2
S5
S7
1.0
Multi
0.0
1.0
Single
(c) 0s
S1
S2
S5
S7
Cluster 1
0.-
Cluster 2
0.2
Figure S.13: EBA clustering for S1/S2/S5/S7. (a) EBA clusters. (b) Voxels which mention just single person and those that mention multiple people. (c) Top nouns. Note that clustering was performed jointly on S1/S2/S5/S7.
Single
Multiple
EBA-1
EBA-2
EBA-1
EBA-2
S1
21.8
68.5
78.2
31.5
S2
31.5
69.5
68.6
30.5
S5
28.8
75.2
71.2
24.8
S7
29.0
63.8
71.0
36.2
Mean
27.8
69.3
72.3
30.8
Table S.2: Distribution of single/multi-person voxels within each EBA cluster. After parsing cach voxel's caption, compute the singlc/multi voxels pereentage of all voxels in the cluster that mention "person" class. We observe that EBA cluster (EBA-I) has higher ratio of voxels that mention multiple people. This reflected both the visualization, the nouns, the human study on ground truth top NSD images each cluster.
29

---

Published as a conference paper at ICLR 2024
A.6 HUMAN STUDY DETAILS
Ten subjects were recruited via prolific.co. These subjects are aged 20 ~ 48; 2 asian, 2 black, white; 5 men, women. For each NSD subject (S1/S2/S5/S7), select the top-100 images for each cluster as ranked by the real average fMRI response. Each of the 100 images were randomly split into 10 non-overlapping subgroups.
Questions were posed in two formats. In the first format, subjects were simultancously presented with images from the two clusters, and select the set where attribute was more prominent, possible answers include cluster- 1/cluster-2/san The second format asked subjects to evaluate set of image from single cluster, and answer yes/no on if an attributeoojeetttyye was present in most of the images.
For the human study results in section 4.4, human evaluator would perform 40 comparisons, from 10 splits and the 4 NSD subjects; with 10 human evaluators per question. We collected 1600 total responses for the four questions in the main text.
For the human study results below, human evaluator would perform 16 judgements, from 4 splits and the NSD subjects; with human evaluators per question; across the clusters of images. We collected 2560 total responses for the eight questions below.
Due to space constraints, we present the single set attribute evaluation (second format described above) results here in the appendix. We divide the results into two tables for presentation purposes.
Are most images...
social
sports
large-scale scene
animals
S1
$2
S5
S7
S1
52
55
S7
S1
52
55
87
S1
S2
S5
S7
EBA-1 EBA-2
88
80
85
85
90
85
88
100
80
85
83
85
20
20
18
23
28
23
35
45
28
25
30
50
38
28
33
60
30
30
30
28
Table S.3: Human study on EBA clustering, first of image attributes. Each human study subject was asked to evaluate groups of 10 images, and answer yes/no on if an attribute was present in most images. Units are in %.
Are most images...
artificial objs
body parts
human faces
multi person
S1
S2
S5
S7
S1
S2
S5
S7
S1
S2
S5
S7
S1
82
S5
S7
EBA-1 LBA-2
78
78
80
75
73
80
78
83
85
78
75
75
100
60
100
85
85
75
83
80
35
30
40
55
28
20
15
45
23
8
18
25
Table S.4: Human study on EBA clustering, second set of image attributes. Each human study subject was asked to evaluate groups of images, and answer yes/no on if an attribute was present in most images. Units are in %.
30

---

Published as a conference paper at ICLR 2024
A.7 TRAINING AND INFERENCE DETAILS
We perform our experiments on a mixture of Nvidia V100 (16GB and 32GB variants), 4090, and 2080 Ti cards. Network training code was implemented using pytorch. Generating one caption for every voxel in higher visual cortex (20, 000+ voxels) in single subject can be completed in less than an hour on a 4090. Compared to brainDiVE the same V100 GPU type, caption based image synthesis with 50 diffusion steps can be done in < 3 seconds, compared to their gradient based approach 25 30 seconds.
For the encoder training, we use the Adam optimizer with decoupled weight decay set to 2c 2. Initial learning rate set 3 and decays exponentially 1.5e 4 over the 100 training epochs. We train each subject independently. The CLIP ViT-B/32 backbone is executed in half-precision (fp16) mode.
During training, we resize the image to 224 x 224. Images are augmented by randomly scaling the pixel values between [0.95, 1.05]. followed by normalization using CLIP image mean and variance. Prior to input to the network, the image is randomly offset by up 4 pixels along either axis, with the empty pixels filled in with edge padding. A small amount of normal noise with = 0, o² = 0.05 independely added to each pixel.
During softmax projection, we set the temperature parameter to 1/150. We observe higher cosine similarity between pre- and post- projection vectors with lower temperatures, but going even lower causes numerical issues. Captions are generated using beam search with a beam width of 5. A set of 2 million images used for the projection, and repeat this with 5 sets. We select the best of 5 by measuing the CLIP similarity between the caption and the IMRI weights using the original encoder. Sentences converted lower case, and further stripped of leading and trailing spaces for analysis.
31

---

Published as a conference paper at ICLR 2024
A.8
TOP ADJECTIVES AND MORE SENTENCES
(a] ba
Faces
Places
Bodies
Food
Words
S1
(b)
A painting of woman with
view of kitchen through window
and woman
close of sliced fruit A close of a clock a building
dancing an the floor
on a plate
A holding trophy
scenic view lake and bench
waman kite next to man
table topped with different types of food
A man halding and sign
kitchen group people
colorful displey food is shown on table
close logo for company
poses for picture
dining room
standing in room
Top adjectives. (a) We extract the most frequent adjectives in each category selective region, note how the adjectives are related to the semantic category in the brain. (b) Additional example sentences for each region. The category selective regions identified via official NSD) functional localizer experiments with a different stimulus set.
Figure S.14:
32

---

Published as a conference paper at ICLR 2024
A.9
ENCODER HITTING STABILITY
(a)
(b)
Max pairwise cosine distance
Average non-self cosine similarity
0.0
0.4
0.0
1.0
Figure S.15: Result on 10-fold aross-validation. (a) measure the cosine distance of the voxel- wise weights across 10-folds. Visualized is the maximum any-pair voxel-wise distance. We find the average maximum any-pair across voxels is 0.02. (b) Average non-self pair-wise cosine similarity across the 10-folds. Note that for each fold, we randomly initialize the weights with kaiming uniform. We find that the fitting process is stable across repeats with an average non-self cosine similarity of 0.98.
33

---

Published as a conference paper at ICLR 2024
Figure S.16: Projection of the 10-fold cross-validation encoders. We perform UMAP projection using the basis from the main paper on each of the 10 encoder weights. We find that aside from minor differences in the FFA/food intersection on the right hemisphere, the large-scale distribution is similar.
34

---

Published as a conference paper at ICLR 2024
A.10 GROUND TRUTH FUNCTIONAL LOCALIZER CATEGORY DISTRIBUTION
EEA
Faces
Places
Bodies
Words
t-value
8
Figure S.17: Ground fruth t-statistic from functional localizer experiments. We plot the ground truth functional localizer result t-statistics. The official functional localizer results are provided by NSD, and are collected using the Stanford VPNL floc dataset. Here red indicates a region which is activated by images from a category. This plot shows the broad category selectivity present in the high order visual areas.
35

---

Published as a conference paper at ICLR 2024
A.11 FINE-GRAINED CONCEPT DISTRIBUTION OUTSIDE EBA
(a) hair
piople
person
FaEQUE
SUt dinss
couple
body
grr
hand
game
plate
food
cake
helrsuz
momman
beciground
dross
Eouple Paintin
group
ai
kitch
1ood
bridge
towerr
baceground
Scy
basetrol
glass
collpiaae
cake
drews
har
hassbal
UMAP1
Ioweer
UMMP
persion
UMAP2
background
palntingg
mountain
3/r
6d1d
bui ding
bridge
kotchon
window
101 Dimmune
body
cellpone
pict.re
SC5SOr
bridge
mull
gless
Taver
Snk
Kimituue
E301
bed
train
patniing
street
builing
kitceen
window
clock
group
Snk
geme
g/1
food
body
kVoww
fiver
bed
(b)
UMAPI
Superio
Superior
CPS
UMAP2
UMAPI
EBA
FBA
Anterior
Anterior
(c)
Figure S.18: Additional UMAP visualizations. Here plot the UMAP dimensionality reduction, and identify the indoor/outdoor concept split in OPA using the 4th UMAP component. Note the Indoor (orange) and Outdoor (purple) gradient along the anterior-posterior axis.
36

---

Published as a conference paper at ICLR 2024
A.12 NORM OF THE EMBEDDINGS WITH AND WITHOUT DECOUPLED PROJECTION
Norm of natural images
Norm with decoupled projection
Norm with coupled projection
Count
12
Norm value
Figure S.19: Norm of the embedding vectors. In the main paper decouple the projection of the norm and direction. Here we visualize the norm natural image embeddings in orange, the norm of the post-projection weights using decoupled projection in blue, and the of the post-projection weights using coupled norm/direction projection in green. As vectors can cancel each other out, the use of decoupled projection in the main paper yields better distribution alignment.
37


Reviews:
[{"review_id": "U6D1oTXM6W", "reviewer": "Reviewer_aznk", "summary": "This paper proposes to combine fMRI encoding models with pretrained vision-language models to find captions for different voxels in visual cortex. They use this method to perform exploratory data analysis on an existing fMRI data (Natural Scenes Dataset, NSD). They find that the method can reconstruct existing known distinctions in preferences in visual cortex (places, faces, bodies, food, words), and can uncover heretofore unrecognized distinctions in the extrastriate body area and in the social network of the brain.", "questions": "My concerns are not about the soundness of the work–which is fine–but about the appropriateness for publication in a machine learning conference. I don't think that there is much extra work the authors could do to convince me otherwise. I'm open to re-evaluation of the paper's merits if the other reviewers deem it appropriate for this conference. \n\nNevertheless, I do have some minor comments:\n\nPage 2: acitvations -> activations\nPage 3-4: I found Figure 2b and its accompanying justification inscrutable. If the point is that there are no images close to unit sphere of captions, and hence blending (eq. 4) must be used to find something closer to the manifold of natural images, this does a poor job of conveying that, and text would be a better way of communicating that. If there is a different point they're trying to make, the authors should take a few sentences to explain what it is. \nPage 7-8: I found the attempt at quantification conveyed in Figures 6 and Table 1 of dubious relevance. If the point of the method is to find visually coherent images that are easy for a human to understand the gist of, using sample images from the NSD itself would do just as well (e.g. Borowski et al. 2021). If the point is to get a better view of what an entire area is selective to, then it seems BrainDiVE works better. The authors should clearly state what they're trying to convey in these figures and tables, they take up a lot of space in the paper but don't add much, in my opinion.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "Overall, this is an interesting application of pretrained vision-language models for better understanding how the brain is organized. The exploration of fine-grained distinctions in the social network of the brain (section 4.4) is quite convincing, especially given the human evaluation results. The paper is clearly written and the demonstration of the use case would, I believe, be of substantial interest to neuroscientists and cognitive scientists.", "weaknesses": "I don't believe that this submission is well-suited for a machine-learning-focused conference such as ICLR. It uses off-the-shelf, pretrained models to find information about visual cortex, which would be primarily of interest to neuroscientists and cognitive scientists. I cannot find substantial methodological advances here that would be of general interest to a conference aimed at machine learning researchers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k6704HH7eb", "reviewer": "Reviewer_xxME", "summary": "This paper proposes a method to find natural language description that maximally activates fMRI response of a voxel  to assess semantic-selectivity of different brain regions.\n\nKey idea is to first train a linear encoder to predict voxel activations from a pretrained CLIP model. Then  Voxel weights are projected to CLIP embedding space to generate captions from a pretrained text decoder (CLIPCap).\n\nThe proposed approach is validated by comparing the selectivity obtained by the proposed method with brain regions which show selectivity for places, food, bodies, words and faces. The authors perform additional analysis to find a person specific region in body-selective areas demonstrating new scientific discovery from this approach.", "questions": "1. Is the semantic selectivity found here limited by training set (NSD)? Can we expect to see semantic specificity that is present in CLIP training data but not in NSD images?\n2. What was the intuition behind the choice  “A photo of a/an [NOUN]”. Did you consider other prompts ?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "1. Use of pre-trained modules (CLIP, CLIPCap) to generate captions which maximize a voxel’s response (Section 3, Figure 1)\n2. Confirmation of BrainSCUBA’s findings on well-known category selective brain regions (Figure 4,5)\n3. Demonstration of category selectivity through a text to image generation model. Figure 5 and Figure 6 show how this method can generate maximally activating images 10 times faster than gradient based maximization (BrainDIVE). The images generated using BrainSCUBA are also more human-interpretable as compared to BrainDIVE.\n4. Finding person and non-person specific cluster within EBA  (Table 3, Figure 8)\n5. Overall the paper is well written and easy to follow. The approach is presented in a simple yet comprehensive manner thus making it easier for new readers to follow.\n6. The approach is validated both qualitatively (figure 5) and quantitatively using different metrics (Figure 6, Table 1,2)\n7. The approach has potential to be extended to investigating more complex semantic specificity which are not feasible using category selective images only.", "weaknesses": "1. Minor: In Figure 3, color code used is not shown in the legend but is there in text in page 6. I recommend to add legend in the Figure also for clarity.\n2. I believe this paper does not fully leverage the potential of BrainSCUBA. The captions generated are currently restricted to Nouns. Semantic selectivity using images is limiting as we have to find natural images that consist of only one concept without confounds. BrainSCUBA can allow a deeper investigation of social interaction through verbs , subject-object pairs and finding which regions are selective for specific interactions/emotions. I emphasize that I mentioned this in the weakness section as I would have loved to see more new results of semantic specificity (other than confirmations).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "k8d2bt1mDC", "reviewer": "Reviewer_79Ny", "summary": "The paper presents a data-driven method, BrainSCUBA, to generate natural language descriptions for images that are predicted to maximally activate individual voxels. This is achieved by first training a neural encoding model to predict the fMRI response from the image embeddings, then projecting the encoding weights into the representational space in a contrastive vision-language model, and finally producing interpretable captions for individual voxels. The results aligned with the previous findings about semantic selectivity in higher-order visual regions. The authors further demonstrated finer patterns that represent the \"person\" category in the brain.", "questions": "- How does the encoding weight vector $W_i$ change regarding to the model's training procedure? For example, if different splits of training and testing datasets are used, to what extent does the $W_i$ fluctuate? This is concerned because all the following analyses depend on a robust and accurate $W_i$. And for a voxel that has poor prediction accuracy (e.g., correlation=0.2), we don't know how well $W_i$ can be trusted as the representative embedding of that voxel.\n- The projection method is intuitive and smart, but there's no direct validation of its effectiveness. Is there a specific reason for using cosine similarity as the distance metric?\n- The map in Fig.3 seems to be very categorical compared to the continuous and distributed brain map resulting from natural speech data (Huth et al. 2016). Does this imply the existence of more separable clusters in higher visual areas? Is there any finer organization within these clusters if you look at more UMAP dimensions, e.g., under the \"human-related\" group (blue voxels) or \"scene-related\" group (pink voxels)?\n- The generated words in Fig. 4 mostly align with expectations. However, certain words, like \"close\" in the \"Faces\" and \"Food\" categories, are strange. Do the authors have an explanation for that?\n- The findings in Fig.7 are interesting. However, I am concerned that the identified regions don't necessarily represent the concept of \"people\". A region that represents another semantic category (for example, \"home\") that often co-occurs with \"people\" words in natural language captions might also be highlighted with this method.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- This method provides more interpretable neural encoding results.\n- The generated text and synthesized images for individual voxels in the higher visual regions align well with previous research findings.", "weaknesses": "- The scientific contribution and significance of this work are unclear. The paper didn't provide much new insights into the neuroscience findings. \n- The method of this paper is not sufficiently evaluated. There are multiple steps in the analysis, including training the encoding model, projecting weights, and transforming them into text or images, each of these steps (except encoding model accuracy) lacks a clear ground truth for comparison. Thus, we can hardly know how much the result deviated from the original voxel representation.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AJl5bBkYWU", "reviewer": "Reviewer_G8Co", "summary": "The authors regressed fMRI neural activities onto CLIP embedding of images, after projecting the weight vector onto the “natural image distribution”, they use the projected weights to generate natural language description and image visualizations.  Using this method, they validated many functional properties of the human visual cortices and defined visual properties for some less charted lands on human cortex.", "questions": "### Questions\n\n- In Eq. 4 why do you choose to use score to weighted average norm and direction separately, instead of averaging the vectors themselves? I can see arguments for both ways, but why do you choose this one?\n- In Fig. 5, the Word Voxel visualizations using BrainSCUBA is always some round cake like object with text on it —— which is kind of strange, while the previous method (BrainDIVE) and NSD samples don’t have this feature. Where do you think this bias could come from?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- This is a fast-paced field. The authors spend the time to add comprehensive references to previous literature, which form a strong foundation for evaluating this work.\n- Simple but principled method. Noticing and addressing the modality gap with embedding projection seems like a key advance to make this work.\n- The close comparison to BrainDiVE is interesting, which kind of suggests that the text caption captures the “essence” of high activation images, without the need for direct gradient ascent, i.e. the selectivity at least for the activation maximizing images is compressed in the words.\n- The authors showed additional applications for neuroscience discovery which is super cool. This visualization/caption tool will help define the visual property of many uncharted lands. Evaluating the separation of clusters with human subjects is convincing.\n- I can see this work’s approach is broadly applicable to multiple neuroscience modalities, e.g. ephys or imaging data from animals. Though language caption may not be a proper medium for them.", "weaknesses": "- Not much a weakness but more like a c**omment.** I think it seems common in this domain (e.g. neuron guided image generation), to pipe together several large-scale pre-trained multi-modal models with brain data and then train linear adaptors between them and then it will work.  So not quite sure how technical challenging this work is comparing to previous ones.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)