Brain Mapping with Dense Features: Grounding Cortical Semantic Selectivity in Natural Images With Vision Transformers
Authors: Andrew Luo
Jacob Yeung
Rushikesh Zawar
Shaurya Rajat Dewan
Margaret Marie Henderson
Leila Wehbe
Michael J. Tarr
Venue: ICLR 2025 Poster (2025)
Keywords: fMRI
visual cortex
neuroscience
cognitive science
brain
vision transformer
semantic selectivity
Primary area: applications to neuroscience & cognitive science
TL;DR: We propose an efficient semantic distillation module and leverage ViTs to investigate selectivity in human visual cortex.

Abstract: We introduce BrainSAIL (Semantic Attribution and Image Localization), a method for linking neural selectivity with spatially distributed semantic visual concepts in natural scenes. BrainSAIL leverages recent advances in large-scale artificial neural networks, using them to provide insights into the functional topology of the brain. To overcome the challenge presented by the co-occurrence of multiple categories in natural images, BrainSAIL exploits semantically consistent, dense spatial features from pre-trained vision models, building upon their demonstrated ability to robustly predict neural activity. This method derives clean, spatially dense embeddings without requiring any additional training, and employs a novel denoising process that leverages the semantic consistency of images under random augmentations. By unifying the space of whole-image embeddings and dense visual features and then applying voxel-wise encoding models to these features, we enable the identification of specific subregions of each image which drive selectivity patterns in different areas of the higher visual cortex. This provides a powerful tool for dissecting the neural mechanisms that underlie semantic visual processing for natural images. We validate BrainSAIL on cortical regions with known category selectivity, demonstrating its ability to accurately localize and disentangle selectivity to diverse visual concepts. Next, we demonstrate BrainSAIL's ability to characterize high-level visual selectivity to scene properties and low-level visual features such as depth, luminance, and saturation, providing insights into the encoding of complex visual information. Finally, we use BrainSAIL to directly compare the feature selectivity of different brain encoding models across different regions of interest in visual cortex. Our innovative method paves the way for significant advances in mapping and decomposing high-level visual representations in the human brain.

Published as a conference paper at ICLR 2025
BRAIN MAPPING WITH DENSE FEATURES: GROUND- ING CORTICAL SEMANTIC SELECTIVITY IN NATURAL IMAGES WITH VISION TRANSFORMERS
Andrew F. Luo 1,2
Jacob Yeung
Rushikesh Zawar'
Shaurya Dewan
Margaret M. Henderson
Leila Wehbe#1
Michael J. Tarr#1
1 Carnegie Mellon University
The University of Hong Kong
aluo@hku.hk, [jacobyeung, szawar, srdewar, amhender, lwehbe, @dchaeelarrr/8mmm edu
ABSTRACT
We introduce BrainSAIL (Semantic Attribution and Image Localization), method for linking neural selectivity with spatially distributed semantic visual concepts natural scenes. BrainSAIL leverages recent advances in large-scale artificial neural networks, using them to provide insights into the functional topology of the brain. To overcome the challenge presented by the co-occurrence of multiple categories in natural images, BrainSAlL exploits semantically consistent, dense spatial features from pre-trained vision models, building upon their demonstrated ability to robustly predict neural activity. This method derives clean, spatially dense embeddings without requiring any additional training, and employs novel denoising process that leverages the semantic consistency of images under random augmentations. By unifying the space of whole-image embeddings and dense visual features and then applying voxel-wise encoding models to these features, we enable the identification of specific subregions of each image which drive selectivity patterns different areas of the higher visual cortex. This provides powerful tool for dissecting the neural mechanisms that underlic semantic visual processing for natural images. We validate BrainSAIL on cortical regions with known category selectivity, demon- strating its ability to accurately localize and disentangle selectivity to diverse visual concepts. Next, we demonstrate BrainSAIL's ability characterize high-level visual selectivity to scene properties and low-level visual features such as depth, luminance, and saturation, providing insights into the encoding of complex visual information. Finally, we use BrainSAIL to directly compare the feature selectivity of different brain encoding models across different regions of interest in visual cortex. Our innovative method paves the way for significant ad advances in mapping and decomposing high-level visual representations in the human brain.
INTRODUCTION
Understanding how the human brain processes and represents visual information from natural experience is a fundamental challenge in neuroscience. The vast majority of our knowledge of the visual system comes from tightly controlled experiments using simplified, hand-crafted images or, at best, real-world photographs of objects against noise backgrounds. Although this paradigm has revealed pattern of preferential neural responses to semantic categories such as faces, places, bodies, words, objects, and food (Sergent et al., 1992; Allison al., 1994; McCarthy et al., 1997; Kanwisher et al., 1997; Aguirre et al., 1996; Epstein & Kanwisher, 1998; Downing et al., 2001; Grill-Spector, 2003; Malach et al., 1995; Khosla al., 2022; Pennock al., 2023; Jain et al., 2023), the visual world we actually experience consists of rich, complex scenes containing many co-occurrigg objects, textures, and contextual associations (Simoncelli & Olshausen, 2001; Torralba & Oliva, 2003). As such, using minimal or single-object stimuli narrows the space of hypothesis testing and limits the ecological relevance of any conclusions, leaving us with an incomplete characterization of how the brain represents and processes real-world visual stimuli.
Co-corresponding authors.

---

Published as a conference paper at ICLR 2025
Recent developments in computer vision models trained on web-scale datasets have enabled learning rich multimodal representations that capture semantic concepts in a suman-aligned manner (Conwell et al., 2022; Wang et al.,, 2023a). In this work, we introduce novel methodology that leverages the power of such models to decompose selectivity patterns in visual cortex by analyzing responses to dense, localized semantic features present in naturalistic images: Semantic Autribution and Image Localization ( ("BrainSAIL"), BrainSAIL allows us to isolate the specific image regions that activate different cortical areas when viewing naturalistic scenes. This method allows us focus on selectivity within complex naturalistic images, thereby enabling a richer decomposition grounded in the full semantic complexity of natural visual experiences.
The core of BrainSAIL involves extracting spatially dense semantic embeddings from images using state-of-the-art models such as CLIPP, DINO, or SigI.IP (Radford et al., 2021; Caron et al., 2021; Zhai et al., 2023). These embeddings bridge the traditionally disparate domains of raw vision data, dense deep semantic features, and measured neural responses. Within this rich embedding space, we can isolate and identify the specific visual features and corresponding image regions that drive selectivity effects in different cortical areas during perception of naturalistic visual scenes. By concurrently modeling localized semantic information, high-level semantic categories, and observed brain activity patterns, BrainSAIL can tease apart the image-level visual drivers of neural tuning preferences across higher visual areas. We validate this dense feature mapping method consisting of human participants viewing many thousands of diverse natural images that span a wide range of semantic categories and visual statisties (Allen et al., 2022).
large-scale fMRI dataset
BrainS AIL's dense embedding framework offers an interpretable view of feature representations across visual regions of the brain. Critically, this view explicitly grounds neural selectivity to localized semantic characteristics inherent in real-world visual experiences. First, we demonstrate the utility of our model for natural images applied to known we show that our model can be used to identify the preference of brain regions sensitive to scene statistics. Finally, we use our model to compare and contrast the feature selectivity for different vision foundation models. In sum, the dense semantic grounding realized in BrainSAll. enables exciting new directions towards understanding and modeling high-level visual representation in humans.
eerggory-sllecivee regions of the cortex. Second,
2 RELATED WORK
A growing body of work leveraging computational modeling and machine learning has explored semantic representation in the higher visual cortex. Approaches include generative image mod- els (Ratan Murty et al., 2021; Gu et al., 2022; Pierzchlewicz et al., 2023; Luo et al., 2023; 2024) and the decoding of visual stimuli (Takagi & Nishimoto, 2022; Chen et al., 2022; Doerig et al., 2022; Ferrante et al., 2023; Liu et al., 2023; Scotti et al., 2024; Yeung et al., 2024). These diverse studies are united by their consideration of the stimulus image as a whole, primarily focusing on the global information contained within the image rather than the individual scene components. In contrast, the method we introduce decomposes an image into its semantic components, enabling the identification of individual, semantically meaningful activating concepts within complex natural images.
Semantic Representation in the Visual Cortex. Using hand-crafted image stimuli, functional mapping studies have identified regions in the human brain that respond preferentially to stimuli representing distinct semantic concepts such as faces, places. bodies, words, objects, and food (Desi- mone et al., 1984; Sergent et al., 1992; Allison et al., 1994; McCarthy et al., 1997; Kanwisher et al., 1997; Gauthier & Tarr, 1997; Aguirre et al., 1996; Epstein & Kanwisher, 1998; Aguirre et al., 1998; O' O'Craven& & Kanwisher, 2000; Nakamura et al., 2000; Aminoff et al., 2007: Downing et al., 2001:
Cohen et al., 2000; Grill Specttrpeetor, 2003; Khosla et al., 2022; Pennock et al., 2023; Jain et al., 2023). One limitation of this simplified approach is that it may not fully capture the contextual complexity of natural vision (Gallant et al., 1998; Mahon, 2022). Addressing this concern, recent work on image-computable encoders has enabled computational tests of visual selectivity using naturalistic images (Naselaris et al., 2011: 1; Huth al., 2012; Yamins et al., 2014; Eickenberg et al., 2017; Wen et al., 2018; Kubilius et al., 2019; Popham et al., 2021; Conwell et al., 2022; Wang et al., 2023a; Luo al., 2023; Prince et al., 2023; Adeli al., 2023; Luo al., 2024: Yang et al., 2024b;a; Efird et al., 2024). Building on this work, our method leverages state-of-the-ar art brain encoding backbones based on vision transformers (Dosovitskiy et al., 2020; Wang al., 2023a) to further explore finer-grained semantic representation in visual cortex.
Visual Contrastive Representation Learning. Self- or weakly-supervee vision models that use contrastive (Xing et al., 2002; Schultz & Joachims, 2003; Chopra et al., 2005; Sohn, 2016 Wu et al.,
2

---

Published as a conference paper at ICLR 2025
(a)
Voxel-wise adapters
Backbone
Visual Stimuli
Frozen
Trained
Brain fMRI
(b)
Repeat l=1...
T(I,C,8,
Backbone
Denoise
C.
Activation
Image I
Learning-Free
Dense
Voxel
(c)
Distillation Module
Adapter
Distilled Feats
Raw Feats
Palm
Sand
Figure 1: The BrainSAll. framework leverages dense visual features. (a) An fMRI encoder learns map from images to voxel-wise activations the brain. Encoders leveraging frozen foundation models based on vision transformers (ViTs) with voxel-wise adapters are currently the highest accuracy models for brain prediction (Conwell et al., 2022; Wang al., 2023a). (b) Given an image and a ViT backbone for the fMRI encoder, we modify the backbone to output dense features. The dense backbone is wrapped inside of Learning-Frrs image I and 27) image coordinates C, and generates transformed images and coordinates (I,, C,) for given transform 0g. The dense features and transformed coordinates are provided to a denoising module to generate clean dense features. The frozen voxel-wise adapter from (a) is applied to each patch to generate dense relevance maps which highlight the image regions activating the voxel. (c) Using CLIP ViT-B/15 6 with the latest NACLIP adapter, show relevance maps using the CLIP text encoder. The NACLIP raw features are highly noisy and contain artifacts, while the distilled features are localized the relevant semantic components with high accuracy. Note that we achieve state-of-the-art open vocabulary CLIP-based segmentation results using our method.
Distillation Module. This module takes an
2018; Musgrave et al., 2020) and masked prediction objectives (Pathak et al., 2016; Kolesnikov et al., 2019; Chen et al., 2020; Zhao et al., 2021: Li et al., 2021; Zhou et al., 2021) are scalable and can be trained massive, diverse datasets to achieve high zero-shot performance
downstream tasks. Contrastive models such as CLIP, DINO, and SigLIP demonstrate strong classification performance without further fine-tuning (Radford al., 2021; Caron al., 2021: Oquab et al., 2023; Zhai et al., 2023). Models that jointly train on language and vision (CLIP/SigLIPP can also classify images using text-based descriptions without fine-tuning. Interestingly, this high level of performance is mirrored in the fact that contrastive models show high performance for predicting neural responses in visual cortex when paired with linear probes (Conwell et al., 2022; Wang et al., 2023a).
Exploring the Brain with Foundation Models. There has been strong interest in leveraging gener- ative models for decoding (reconstructing) visual stimuli conditioned brain activations (Kamitani & Tong, 2005; Han et al., 2019; Seeliger et al., 2018; Shen et al., 2019; Ren et al., 2021; Takagi & Nishimoto, 2022: Chen al., 2023; Lu al., 2023; Ozcelik VanRullen, 2023; Doerig al., 2022; Ferrante et al., 2023; Liu et al., 2023; Mai & Zhang, 2023; Scotti et al., 2024). A related approach generates novel stimuli that are posited to best to activate target brain region (as opposed to reconstructing the original stimulus) (Walker et al., 2019; Bashivan et al., 2019) with recent attempts utilizing GANs or Diffusion models to constrain the synthesized output (Ponce et al., 2019; Ratan Murty et al., 2021; Gu et al., 2022; Luo et al., 2023; 2024). While these models have shown positive results, they all rely on images as a whole, whereas BrainSAIL. seeks disentangle complex images into their semantically meaningful components and localize those parts of the image that elicit activation for different brain voxels or regions.
METHODS
Our aim is to generate spatial attribution maps for arbitrary voxels in the higher visual cortex. Unlike the early visual cortex, which is believed to be primarily selective for "simple features" (Stork &
3

---

Published as a conference paper at ICLR 2025
Wilson, 1990), the higher visual cortex exhibits semantic selectivity pattern that, at present, is best predicted by deep networks (Conwell et al., 2022; Wang et al., 2023a). As illustrated in Figure 1, to create spatial attributions maps for brain voxels, we first train voxel-wise fMRI encoders to map images to brain activations. Second, we derive dense features from pre-trained vision transformers (ViT) used as the backbone for these encoders. Third, we demonstrate that an artifact-free dense feature map can be derived for high-throughput exploration of selectivity with the visual cortex.
3.1 IMAGE-TO-BRAN ENCODERS FOR THE HIGHER VISUAL CORTEX
voxel-wise image-computable fMRI encoder is a model F& that predicts fMRI activations (betas) for E RIXN where N represents the number of voxels in the brain. The encoder is conditioned on image input I E HR x3 where Fg = B. Recent work has demonstrated that encoders that rely on features extracted from large vision foundation models achieve excellent predictive performance, where higher visual cortex is best predicted by deeper layers in the model (Wang et al., 2023a). In this setting, the backbone model is usually frozen, while a per-voxel adapter typically parameterized linear layer is trained to map from network features voxel activations. In that we focus on the higher visual cortex exclusively, we utilize a two component design for our encoder: (1) frozen vision foundation model backbone G(Z) which outputs R dimension embedding vector for each image; (2) a per-voxel adapter parameterized as a linear probe with weight W e R RM N and bias b R!X L which takes as input a unit-norm image embedding.
Ging (I)
X W
+ b = +B
(1)
Ging (2)
It should be noted that BrainSAII. is not restricted to linear probes, and can work with arbitrary voxel- wise parameterizations, including MLPss. Linear probes are used here as they are widely adopted in IMRI encoder literature and empirically achieve good performance. BrainSAIL is compatible with any Vision Transformer (ViT)-based model, making it readily applicable to the vast majority of modern visual foundation models which predominantly employ ViT architectures. Additional results are presented in the supplemental. We train our model with MSE loss, and evaluate the encoder on the test set. In Figure we show that our encoder achieves state-of- mate-offtheearr R².
3.2 DERIVING DENSE FEATURES FROM VIT BACKBONES
The emergence of vision models trained on contrastive image-text objective has fueled interest in zero-shot open-vocabulary image classification methods. For example, CLIP has shown that images can be classified without foreknowledge of the test time classes during training; instead the category of interest can be described using language during test time. Of late, this capabil- ity has been extended from classification to segmentation. Compared to methods that require human annotation (Li et al., 2022) and perform poorly on oft-oo-diistrbuttion images (Jatavallab- hula et al., 2023; Kerr et al., 2023), these new methods require further training and directly extract dense features that lie in the same space as the image/text embedding. These dense fea- ture extraction methods operate by modifying the last self-attention (SA) block within the typical ViT architecture (MaskCLIP, Zhou et al. (2022); SCLIP, Wang et al. (2023b); NACLIP, Hajimiri al. (2024)). For vision models of this sort trained on contrastive objective, the output is composed of a single [CLS] token, which is supervised using contrastive loss; and numer- ous patch tokens which correspond to specific spatial locations. Let (gi, k, v2) be the query, key, value features respectively for a single image patch i, with a total of m spatial patches. For a given patch j at the final layer, where f denotes any function applied to the [CLS) after the last self-attention, the [CLS] token and each dense token is a convex combination of v features:
Out Orig, [
softmax 9.A. AT
Out Musk, f(y)) Out NA,
softmax 9 + FWly sy (2)
MaskCLIP directly removes the convex re- re-geithting and outputs the value feature for each patch token directly. SCLIP and NACLIP reintroduce the weighting to reduce output artifacts, but modify it with correlative self-attention (CSA); by using CSA with spatial attentive bias w. Here, we utilize NACLIP as the dense adaptor for CLIP. The other two backbones in Section 4.4 use an updated ViT architecture with "register tokens" (Darcet et al., 2023). As these have not been explored in the context of CSA, we utilize MaskCLIP as the dense adaptor. We treat dense features as unit-norm.
4

---

Published as a conference paper at ICLR 2025
(b)
Image I
I V.p
I (2
Raw features
Distilled features
u-coord
(c)
1a
Low
High
v-cgord
C₂ Flower/Leaf Bird Raw features
Flower/leaf Bird
C, Figure 2: The Learning- Free Distillation Module. (a) Given an image, we generate image-space co- ordinates (u, v) for each pixel. We then randomly sample from OI... where 0; has vertical/horizontal offset, and left-right flips. The augmented images are provided to frozen backbone with dense adapter. The features are projected to the original image space via an inverse transform T T (0.). (b) UMAP visualization of the dense features. The same fitted basis is used for both visualizations. (c) With CLIP, we can perform zero-shot text queries. Note the artifacts above the bird's head. In practice artifact location is different for each image. The distilled results are significantly better.
Coordinates
Distilled features
3.3 LEARNING-FREE FEATURE DISTILLATION
As only the [CLS] is supervised in these contrastive models, as shown Figures and 2, the extracted dense embeddings often have artifacts even when using the latest NACLIP method which seeks to reduce artifacts. While methods such Darcet et al. (2023) improve spatial consistency via architec- tural improvements, they require training the model
Input: Image Z:
Image space coordinates C;
Augmentation parameters 01...
Augmentation function T:
ViT model with dense adapter M;
1. Zero init clean feature tensor Q
Zero init count tensor K
with architecture modifications that are computa-
3: For in {I...n}:
tionally costly. Methods like Kobayashi al. (2022)
4.
0; = (wi, Us flip,)
rely on expensive gradient-based optimization us-
(I,CC,) TII, C.
ing MSE loss over multiple views in 3D. Conse-
6.
Dense feature F; = M
quently, in order to facilitate high-throughput charac-
(Fyalid Cyalid) = TT- 0.)
terization the visual cortex over large datasets, we propose an efficient learning-free distillation mod- ule. Given an image Z, we first generate n aug- mentation parameters 01...., where 02 consists of a horizontal/vertien
8.
Q[Cyalid) Qu + 1 poyalid
9.
K K|Cralid K ((yyalid)
10.
return Q/K
Algo Learning-Free Feature Distillation
offset (u, 02) and horizontal flip, E {0, }.. We further generate the image space coordinates C = (u-coord, v-coord), where u E [0, 1] goes from left-to-right, while e E [0. 1] goes top-to-bottom. We describe our full transform generate 2D "views". Our method distills clean semantic map, as visual semantics are equivariant shift and horizontal flips. We note that averaging over the number of augmentation is extracting an optimal embedding under squared error (squared euclidean). Let p" be the optimal embedding under MSE for a patch, and p. with i {1...n} be the feature candidates under image augmentation:
Algorithm 1. These augmentations effectively
1" min p
Σ p
P11
min [|
211 + + 11 -012)
(3)
min
pi pi 2pi T レ p +
pn pn - 2ph T 1
omitting pi p
(4)
P
min P
n n
2 1 p)
min
n n. TT D- 2n > (11/n) Di
(5)
1=1
The objective can be expressed as || (1/n) 1 > 0, then p = (1/n)
In Table 1, we compare pre- and post- smoothing results. Under the Pearson metric, which does not assume prior category knowledge, smoothing yields the best performance three of four datasets. We apply the voxel-wise adapters the per patch dense features derive the final relevance map.
5

---

Published as a conference paper at ICLR 2025
ADE20K
COCO Object
COCO Stuff
VOC20
mloUt
Pearsont
mloUt
Pearsont
mloUt
Pearson
mloUT
Pearson
SCLIP
16,45
0.308
33.52
0.353
21.95
0.309
81.54
0.551
NACLIP
17.69
0.425
33.14
0.418
22.58
0.393
77.09
0.473
+Distilled
18.19
0.443
34.15
0.435
23.08
0.405
79.09
0.489
Table Open-vocabulary segmentation with dense CLIP features. We validate the effectiveness of our learning- free smoothing approach on segmentation datasets in a zero-shot setting (without any training). This performance is state-of-the-art for open vocabulary segmentation. Note mloU scores are multiplied by 100. Our smoothing improves the results.
UIMAP3
Superior
Superior
UMAP2
GIAAP1
Food Toaion
Tigho
Right Hemisphere
Left Hemisphere
(b)
(c)
Woman
Food
Food
Furniture
Man
Opera House
(d)
Figure 3: Joint dimensional reduction of higher visual cortex encoder weights and images using BrainSAIL. We use UMAP to perform visualization of the encoder weights. This same UMAP basis is reused for images. (a) Cortical flatmap of S1. Note that the overlaid white region outlines and labels visualized UMAP results. (b) Embeddings from novel images are computed with BrainSAIL and transformed using the fMRI UMAP. For each quartet of images, the content is follows Top left: Original RGB image: Top right: Dimension reduction of BrainSAIL embeddings for the image; Bottom: Two text queries using CLIP text branch showing ianguage-indicatte relevance results. (c) UMAP results on inflated view of the brain for S1. (d) UMAP results on cortical flatmaps for $2, S5 and S7. These results demonstrate that BrainSAIL can effectively localize semantically meaningful components of natural images and map them to appropriate brain regions. The cortical maps show color-coded mappings that align well with dunctionally-defii (EBA), face regions (FFA/aTL-facee Note that the food regions have been identified as flanking FFA by Jain et al. (2023), but we do not have independent functional localizer data for food for these subjects
derived from functional localizer data collected independently from the
regions: body regions place regions (RSC/OPA/PPA), and food regions (yellow).

---

Published as a conference paper at ICLR 2025
Region
Faces
Places
Bodies
Words
Food
S1
S2
S5
S7
S1
S2
S7
S1
S2
S7
S2
S7
S1
S2
S5
Face
46
48
54
40
1
1
2
2
12
11
12
13
11
12
11
1
2
1
Places
1
76
80
89
75
2
2
12
9
8
10
Bodies
26
16
21
27
5
1
0
1
50
41
55
48
15
13
21
30
9
5
1
5
Words 1
7
3
8
6
3
2
9
6
7
7
38
28
26
23
16
13
17
35
Food 26
21
22
12
15
13
40
23
30
36
30
27
66
73
71
Table 2: CLIP text alignment for each category selective brain region. For each category selective
brain region, we take the top-100 images from the NSD test set that elicit the highest fMRI response
for each region. We then use BrainSAIL to compute the relevance maps for the top-100 images for each region. For each image, additional relevance maps are computed using the CLIP text encoder with text prompts from the five relevant categories. The text prompt with the highest Pearson correlation to the BrainSAll. relevance map is recorded as the category for that image. Units in %.
4 RESULTS
We utilize BrainSAIL to localize the semantic selectivity of different brain regions and demonstrate that the relevance maps are interpretable throughout the brain and correlate well with the known category-selective regions. We then explore the selectivity of higher visual cortex with respect to localized scene structure and image properties. Finally, we compare and contrast the localization results from three different vision foundation models, These results establish BrainS as a novel technique for mapping and understanding the semantics of visual representations in the brain.
4.1 SETUP
We use the Natural Scenes Dataset (NSD; Allen et al. (2022)), the largest 7T fMRI dataset of human visual responses, focusing on four subjects (S1, S2, S5, S7) who viewed the full 10,000 image set (a subset of COCO images) three times each. fMRI activations (betas) were derived using GLMSingle (Prince et al., 2022) and normalized per session (μ+ = 0, 1). Responses to repeated images were averaged. A brain encoder for each subject was trained on ~ 9000 unique images per subject, with the remaining 1000 images viewed by all subjects being used for R² validation the test set. Supplementary results for other subjects are included in the appendix. Face, place, body, and word regions were defined using independent category localizer data from NSD with threshold of t >> 2 (Stigliani et al., 2015). Food regions were defined using masks provided by Jain et al. (2023).
We train three encoders based on different neural network backbones. For all three, we utilize the ViT TATT-aac Base model size. (1) For CLIP, we utilize OpenAl's official ViT-B// 6 weights. This a network trained on an infoNCE contrastive image-text objective. (2) For DINO, we uti- lize the latest official DINOv2 ViT-B/14 111g,, and is network trained on image-only self- supervision (Darcet et al., 2023). (3) For SigLIP, we utilize NVIDIA's implementation based on RADIOV2.5 5 ViT-B/16 (Ranzinger et al., 2024), as the original Google variant used a non- standard architecture. SigLIP utilizes a pairwise non-contrastive image-text objective (Zhai et al., 2023). All IMRI encoders are trained using MSE loss, with the backbone frozen. We validate the test time R22 in Figure and find that we achieve state-of-the-arr art results similar to Wang et al. (2023a) and Luo et al. (2024). We use CLIP for Sections 4.2 and 4.3, it is the most widely used backbone in fMRI literature. We use 51 augmentation steps unless otherwise noted.
4.2 IMAGE FACTORIZATION USING THE BRAIN
To explore how different retpoory- regions in higher visual cortex align to different image parts we apply UMAP (McInnes al., 2018) with an angular metric linear brain weights and apply the same UMAP basis to dense features as produced by BrainSAlL. Note that during dimensionality reduction we do not utilize any cortex category masks from NSD the region of interest outlines on the cortex in Figure 3a are for visualization purposes only and are derived from independent NSD functional localizers. As shown in Figure 3, we find that the factorization of the brain is well aligned to pre-identified functional regions, and broadly segments the cortex into axes along "people", "scenes" and "Food". particular, place regions, including the retrosplenial cortex (RSC), occipital place area (OPA), and parahippocampal place area (PPA), show selectivity for scene components (magenta). People regions, including the extrastriate body area (EBA), fusiform face area (FFA), occipital face area (OFA), show selectivity for face and body parts in the image (Green-Blue). Finally, find that the recently identified food-responsive region that roughly surrounds FFA (Yellow) Khosla et al. (2022); Pennock et al. (2023); Jain et al. (2023) strongly corresponds to food in images. These results
7

---

Published as a conference paper at ICLR 2025
Place Voxels
Word Voxels
NEO
STOP
UNAP (testures
STOP
Face Voxels
Body Voxels
UIMAP Insmures
Food Voxels
NS0
UNNAD 00OTSAA (ncstuncs)
Figure 4: Grounding results using BrainSAlL. We visualize the top test set images as predicted by the CLIP fMRI encoder for each category selective region. For each image, we also visualize the image-wise UMAP for the distilled dense features. Note the UMAP basis here is computed imagewise, and not shared with Figure 3. The image-wise UMAP shows the semantic components present in the dense features. For each image, we further visualize the feature relevancy map for the category selective voxels illustrating that this method extracts the semantically relevant regions in complex compositional images.
establish that BrainSAIL can be used to characterize higher-level selectivity to individual semantic categories in complex natural images without prior knowledge of their semantic selectivity.
We further quantify the feature relevance maps for broad category selective regions in Figure 4 and Table We use the brain encoder to predict the top-5 images for the place/word/face/bodd regions, and the top-10 10 images for the food-responsive region. We find that method can effectively localize the objects relevant to each category- selective brain region. Note that the word region is known to have selectivity faces (Mei et al., 2010) and food (Khosla & Wehbe, 2022).
4.3 CORTEX SELECTIVITY TO IMAGE FEATURES
Going beyond semantic categories, seek to explore the low- and mid-level image feature correlates that correspond to different brain regions. Prior work explored this by training a convolutional encoder on each NSD subject, which is limited 10, 000 images each (Sarch al., 2023). One concern is that using a small dataset with convolutional backbone can lead to overfitting to the dataset's specific features and exacerbate the inherent biases of convolutional networks. To address this limitation. our method leverages vision transformers trained on massive datasets of hundreds of millions of images, thereby avoiding the hard-coded inductive biases present in CNNs (Raghu et al., 2021). We visualize BrainSAIL. feature dissection results in Figure 5. Our method can successfully identify the known scene selective regions (RSC/OPA/PPA) as preferring high depth, and is successful even in OPA where Sarch et al. (2023) fails. We believe this is likely because OPA processes higher-level associative content and affordances (Bonner & Epstein, 2017; Aminoff & Tarr, 2021). Similarly, we identify the region surrounding FFA as being selective to high color saturation, which correspond to the responsseon regions identified by Jain et al. (2023) and others. In OPA, we identify a split in color luminance preference, which is similar the indoor/outdoor preferring regions identified
8

---

Published as a conference paper at ICLR 2025
(a)
(b)
(c)
OP
GoA
FFA-1
Food
12 Low
Color
High
Low
Color
High
Depth (meters)
Saturation
Luminance
Figure 5: Feature correlates with BrainSAIL. We visualize the depth, color saturation, and color luminance (brightness) correlates for each brain region using BrainSAIL. (a) The scene selective regions, retrosplenial cortex (RSC), parahippocampal place area (PPA), and occipital place area (OPA) are all identified as having a preference for high depth. (b) On the ventral surface, we identify two stripes each hemisphere, surrounding FFA with high saturation preference. These the same brain regions identified by Jain et al. (2023) as being food selective. (c) In OPA, we identify an anterior/posterior split, where one region has high color luminance preference, and the other has low color luminance preference. This are the same regions identified by Luo et al. (2023) as being outdoor/indoor selective.
(a)
(b)
x=y
SigLIP R2
DINO R
DINO RZ
CLIP
DINO S1
SigLIP
CLIP R
SigLIP R
CLIP R
Figure 6: Comparing the brain prediction performance for different encoder backbones. (a) We validate each encoder R² on a test set and find that all three models achieve very high performance (comparable to Wang et al. (2023a)). (b) The voxel-wise correlation of test set R² for the three models. CLIP and SigLIP, which rely on language supervision, achieve higher performance than DINO (which trained via supervvinoi with images).
by Lescroart & Gallant (2019), Peer et al. (2019), and Luo et al. (2023). These results demonstrate that our method can identify fine-grained selectivity with more broadly characterized brain regions.
4.4 ARE BRAIN ENCODERS EQUIVALENT?
Recent tirh-rerformin models such as CLIP, DINO, and SigLIP differ in their training objectives, architectures, and datasets: CLIP employs contrastive image-language objective, DINO utilizes self-supervised image loss without explicit linguistic guidance, and SigLIP leverages non- contrastive pairwise image-language loss. Despite these differences, when employed as the backbone for fMRI encoders, these models exhibit similar performance in predicting brain responses, achieving comparable R values on the test set as shown Figure This observation raises an important question about the nature of each model's learned features and their alignment with one another: Do these models converge upon similar feature representations (Chen & Bonner, 2024) for category selective brain regions despite their varied training paradigms?
To investigate the representational differences between the models, we perform BrainSAIL analysis for scene-, face-, and food-responsive brain regions and qualitatively visualize the results Figure 8. While all three models exhibit broad similarities in their grounding maps, DINO, trained without language supervision, demonstrates a stronger sensitivity to low-level visual features compared to CLIP and SigLIP (Wang et al., 2023a). This is evident in the cood-responsive region (Figure 8), where DINO's grounding map for a pizza image excludes the toppings and misses differently colored vegetables on metal plate, suggesting focus on color and texture rather than the concept of "food" itself. Similarly, in the face region, DINO's grounding map exhibits less reliance on semantically relevant features such as eyes, nose, and mouth. We hypothesize that this greater sensitivity to visual features in DINO stems from its lack of language guidance during training, preventing from learning the liveer-level semantic correlations that link visually disparate parts and objects within a category. As high-performing "proxy models" of visual brain representation (Leeds et al., 2013), these and other underlying model characteristics architecture, training objective, training dataset,

---

Published as a conference paper at ICLR 2025
Spearman correation.
CLIP & SigLIP
DINO & SigLIP
DINO & CLIP
S1
0.6-
S2
0.0
+ood
*-lorts
VEC
Food
K5C
woodes
Words
Figure 7: Model similarity across ROIs Brain encoder backbone spatial similarity for the ground truth top-100 images from the test set for each category- selectivv brain region. A * denotes a domain- defined network of regions encompassing multiple ROIs. CLIP and SigLIP relevance maps are more similar to another than either is to DINO. Error bars indicate standard error across the 100 images.
Place Voxels
Facc Voxels
CUP
Saup
DINO
Food Voxels
NGB
CUP
Sqgui
DNO
Figure 8: Comparing different brain encoder backbones with BrainSAIL. Visualization of the top test set images for the place, face, and food category-selective selective brain regions as predicted by CLIP, SigLIP, and DINO. While all models show broadly similar feature relevance for given brain area, there are important differences. DINO, with no language supervision, exhibits greater sensitivity to visual similarity, at the cost of semantic coherence.
etc. important considerations for developing more robust encoding models that can bridge the gap between artificial and biological vision systems.
5 DISCUSSION
Limitations and Future Work. BrainSAIL achieves strong localization performance and benefits from pre-trained vision transformer, reducing reliance on the fMRI dataset for backbone training. However, it is still necessary to train the fMRI encoder on these data, and thus potential dataset biases in the human neural data and how it was collected can influence the learned representations and conclusions (Shirakawa et al., 2024). Future work should explore training on larger and more diverse neural datasets mitigate this limitation and enhance the generalizability of our findings.
Conclusion. We propose BrainSAIL, a method that leverages vision foundation models to interro- gate which semantic components of complex natural images lead to the neural activation of specific regions of the brain. Based on the vision transformer architecture, we: (1) semantically attribute and localize relevant objects in complex compositiona images; (2) jointly factorize images and seman- tically selective regions in the human brain; (3) identify the feature correlates of depth, saturation, and luminance that underlie semantic selectivity; (4) explicate differences in fMRI encoders that achieve similar overall brain prediction performance. In toto, these results establish that BrainSAlI. a powerful new approach to data-driven explorations of the human higher visual cortex.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Hossein Adeli, Sun Minni, and Nikolaus Kriegeskorte. Predicting brain activity using transformers. bioRxiv, pp. 2023-08 08, 2023. 2
Geoffrey K Aguirre, John Detre, David C Alsop, and Mark D D'Esposito. The parahippocampus subserves topographical learning in man. Cerebral cortex, 6(6):823-829 1996. 1,2
Geoffrey Aguirre, Eric Zarahn, and Mark D D'Esposito. An area within human ventral cortex sensitive to "building" stimuli: evidence and implications. Neuron, 21(2):373-333, 1998. 2
Emily J Allen, Ghislain Yve,, Yihan Wu, Jesse L. Breedlove, Jacob S Prince, Logan T Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, et al. A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence. Nature neuroscience, 25(1):116-122) 2022. 2,
Truett Allison, Gregory McCarthy, Anna Nobre, Aina Puce, and Aysenil Belger. Human extrastriate visual cortex and the perception of faces, words, numbers, and colors. Cerebral cortex, 4(5): 544-554, 1994. 2
Elissa Aminoff, Nurit Gronau, and Moshe Bar. The parahippocampal cortex mediates spatial and nonspatial associations. Cerebral cortex, 17(7)1493-1503, 2007. 2
Elissa M Aminoff and Michael J Tarr. Functional context affects scene processing. Journal of cogmitive neuroxcience, 33(5):333-94,, 2021.
Pouya Bashivan, Kohitij Kar, and James J DiCarlo. Neural population control via deep image synthesis. Science, 36+(6439))eaav9436 2019. 3
Michael F Bonner and Russell A Epstein. Coding of navigational affordances in the human visual system. Proceedings of the National Academy of Sciences, 114(18):4793 4798, 2017.
Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings the IEEE/CVF international conference computer vision. pp. 9650-9660, 2021. 2,
Mark Chen, Alec Radford, Rewon Child, Jeffrey Wu, Heewoo Jun, David Luan, and Ilya Sutskever. Generative pretraining from pixels. In International conference on machine learning, pp- 1691- 1703. PMLR, 2020. 3
Zijiao Chen, Jiaxin Qing, Tiange Xiang, Wan Lin Yue, and Juan Helen Zhou. Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding. arXiv preprint arXiv:2211.06955 1()):, 2022.
Zijiao Chen, Jiaxin Qing, Tiange Xiang, Wan Lin Yue, and Juan Helen Zhou. Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding. In Proceedings of the IEEF/CVF Conference on Computer Vision and Patlern Recognition, pp. 22710-22720, 2023.
Zirui Chen and Michael F Bonner. Universal dimensions of visual representation. arXiv preprint arXiv:2408. 12804, 2024. 9
Sumit Chopra, Raia Hadsell, and Yann LeCun. Learning a similarity metric discriminatively, with application face verification. In 2005 IEEE computer society conference on computer vision and pattern recognition (CVPR 05), volume 1,, pp. 539-546 IEEE, 2005.
Laurent Cohen, Stanislas Dehaene, Lionel Naccache, Stéphane Lehéricy, Ghislaine Dehaene- Lambertz, Marie-Anne Hénaff, and François Michel. The visual word form area: spatial and temporal characterization of an initial stage of reading in normal subjects and posterior split-brain patients. Brain, 123(2):291-307, 2000. 2
Colin Conwell, Jacob S Prince, Kendrick N Kay, George Alvarez, and Talia Konkle. What 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines? BioRxiv, pp. 2022-03, 2022. 2,
11

---

Published as a conference paper at ICLR 2025
Timothée Darcet, Maxime Oquab, Julien Mairal, and Piotr Bojanowski. Vision transformers need registers. arXiv preprint arXiv:2309.16588 2023. 4,
Robert Desimone, Thomas D Albright, Charles G Gross, and Charles Bruce. ttimulus-selectiig properties inferior temporal neurons in the macaque. Journal of Neuroscience, 4(8):2051-200) 1984. 2
Adrien Doerig, Tim C Kietzmann, Emily Allen, Yihan Wu, Thomas Naselaris, Kendrick Kay, and lan Charest. Semantic scene descriptions objective of human vision. arXiv preprini arXiv:2209. 11737, 2022. 2,3
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition scale. arXiv preprint arXiv:2010.11929, 2020. 2
Paul E. Downing, Yuhong Jiang, Miles Shuman, and Nancy Kanwisher: A cortical area selective for visual processing of the human body. Science, 293455399 2001. 1,2
Cory Efird, Alex Murphy, Joel Zylberberg, and Alona Fyshe. What's the opposite of face? finding shared decodable concepts and their negations in the brain, arXiv preprint arXiv:2405. 17663, 2024.
Michael Eickenberg, Alexandre Gramfort, Gaël Varoquaux, and Bertrand Thirion. Seeing it all: Convolutional network layers map the function of the human visual system. Neurolmage, 152: 184-194, 2017.
Russell Epstein and Nancy Kanwisher. cortical representation of the local visual environment. Nature, 392(6666):598-611) 601, 1998. 1,2
Matteo Ferrante, Furkan Ozcelik, Tommaso Boccato, Rufin VanRullen, and Nicola Toschi. Brain captioning: Decoding human brain activity into images and text. arXiv preprint arXiv:2305./1550, 2023. 2,
Jack L Gallant, Charles E Connor, and David C Van Essen. Neural activity in areas vl, v2 and v4 during free viewing of natural scenes compared controlled viewing. Neuroreport, 9(7): 1673-1678, 1998. 2
Isabel Gauthier and Michael Tarr. Becoming "Greeble" expert: Exploring mechanisms for face recognition. Vision research, 37(12):1633-6622 1997. 2
Kalanit Grill-Spector. The neural basis of object perception. Current opinion neurobiology, 13(2):
159-166, 2003. 1.2
Zijin Gu, Keith Wakefield Jamison, Meenakshi Khosla, Emily Allen, Yihan Wu, Ghislain St- Yves, Thomas Naselaris, Kendrick Kay, Mert R Sabuncu, and Amy Kuceyeski. NeuroGen: activation optimized image synthesis for discovery neuroscience. Neurolmage, 18811, 2022. 2,
Sina Hajimiri, Ismail Ben Ayed, and Jose Dolz. Pay attention to your neighbours: Training-free open- opeablocabylary semantic segmentation. arXiv preprint wrXiv:2404.08735 2024. 4, 19
Kuan Han, Haiguang Wen, Junxing Shi, Kun-Han Lu, Yizhen Zhang, Di Fu, and Zhongming Liu. Variational autoencoder: An unsupervised model for encoding and decoding fmri activity visual cortex. Neurolmage, 198: 188:125-133 136, 2019.
Alexander G Huth, Shinji Nishimoto, An T Vu, and Jack L Gallant. A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron, 76(6): 1210- 1224, 2012. 2
Nidhi Jain, Aria Wang, Margaret M. Henderson, Ruogu Lin, Jacob S. Prince, Michael J. Tarr, and Leila Wehbe. Selectivity for food in human ventral visual cortex. Communications Biology 2023 6:1, 6:1-14, 2 2023. ISSN 2399-3642, doi: 10. 00.1038/42203-003--055555 1, 2, 6, 26
12

---

Published as a conference paper at ICLR 2025
Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Alaa Maalouf, Shuang Li, Ganesh Iyer, Soroush Saryazdi, Nikhil Keetha, et al. Conceptfusion: Open-set multimodal 3d mapping. arXiv preprint DrXXi:2302..7241, 2023. 4
Yukiyasu Kamitani and Frank Tong. Decoding the visual and subjective contents of the human brain. Nature neuroscience, 8(5):679-88,, 2005, 3
Nancy Kanwisher, Josh McDermott, and Marvin M Chun. The fusiform face area: a module in human extrastriate cortex specialized for face perception. Journal of neuroscience, 17(11):400-4411, 1997. 1,2
Justin Kerr, Chung Min Kim, Ken Goldberg, Angjoo Kanazawa, and Matthew Tancik. Lerf: Language embedded radiance fields. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp.
2023. 4, 19
Meenakshi Khosla and Leila Wehbe. High-level visual areas act like domain-general filters with strong selectivity and functional specialization. bioRviv, pp. 2022-03, 2022. 8
Meenakshi Khosla, N. Apurva Ratan Murty, and Nancy Kanwisher. highly selective response to food in human visual cortex revealed by hypothesis-free voxel decomposition. Current Biology, 32:1-13, 2022. 1, 2,
Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. Decomposing nerf for editing via feature field distillation. Advances in Neural Information Processing Systems, 35:23311 233300, 2022. 5, 19
Alexander Kolesnikov, Xiaohua Zhai, and Lucas Beyer. Revisiting self-supervised visual repre- sentation learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1920-1929, 2019. 3
Jonas Kubilius, Martin Schrimpf, Kohitij Kar, Rishi Rajalingham, Ha Hong, Najib Majaj. Elias Issa, Pouya Bashivan, Jonathan Prescott-Roy. Kailyn Schmidt, et al. Brain-like object recognition with high-performing shallow recurrent anns. Advances in neural information processing systems, 32, 2019. 2
Daniel D Leeds, Darren A Seibert, John Pyles, and Michael Tarr. Comparing visual representations across human IMRI and computational vision. J Vis, 13(13)(55)1-27, 2013. 9
Mark D Lescroart and Jack L Gallant. Human scene-selective areas represent 3d configurations of surfaces. Neuron, (01(1):178-192 2019. 9
Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and René Ranftl. Language-driven semantic segmentation. arXiv preprint arXiv:2201.0333 2022. 4, 19
Zhaowen Li, Zhiyang Chen, Fan Yang. Wei Li, Yousong Zhu, Chaoyang Zhao, Rui Deng. Liwei Wu, Rui Zhao, Ming Tang, et al. Mst: Masked self-supervised transformer for visual representation.
Advances in Neural Information Processing Systems, 34:13165-13176, 2021. 3
Yulong Liu, Yongqiang Ma. Wei Zhou, Guibo Zhu, and Nanning Zheng. Brainclip: Bridging brain and visual-linguistic representation via clip for generic natural visual stimulus decoding from fmri. arXiv preprint 312. 2023. 2.
Yizhuo Lu, Changde Du, Dianpeng Wang, and Huiguang He. Minddiffuser: Controlled image reconstruction from human brain activity with semantic and structural diffusion. arXiv preprint arXiv:2303. /4139, 2023. 3
Andrew Luo, Margaret Marie Henderson, Michael J. Tarr, and Leila Wehbe. Brainscuba: Fine- grained natural language captions of visual cortex selectivity. In The Twelfth International Confer- ence on Learning Representations, 2024. URL https:/ wpwp/evieeeeeeeee//e/pp/e. net/forum? mOYHXUUTKU. 3.7
Andrew F Luo, Margaret M Henderson, Leila Wehbe, and Michael J Tarr. Brain diffusion for visual ex- ploration: Cortical discovery using large scale generative models. arXiy preprint XXX:2330600008) 2023. 2, 3,
13

---

Published as a conference paper at ICLR 2025
Bradford Z. Mahon. Domain- connectivity drives the organization of object knowledge in the brain. Handbook of clinical neurology, 187:221-244, 2022. doi: 10.1016B998-00-1-2224429 00028-6. Place: Netherlands. 2
Weijian Mai and Zhijun Zhang. Unibrain: Unify image reconstruction and captioning all in one diffusion model from human brain activity, arXiv preprini arXiv:2308.07422 2023.
Rafael Malach, JB Reppas, RR Benson, KK Kwong, H Jiang, WA Kennedy, PJ Ledden, TJ Brady, BR Rosen, and RB Tootell. Object-related activity revealed by functional magnetic resonance imaging in human occipital cortex. Proceedings of the National Academy of Sciences, 92(18): 8135-81333 1995.
Gregory McCarthy, Aina Puce, John C Gore, and Truett Allison. Face-specific processing in the human fusiform gyrus. Journal of cognitive neuroscience, 9(5):605 1997. 1,2
Leland McInnes, John Healy, and James Melville. Umap: Uniform manifold approximation and projection for dimension reduction. arXiv preprint arXiv: 1002.v:800..3300 082.0222.22 2018. 7
Leilei Mei, Gui Xue, Chuansheng Chen, Feng Xue, Mingxia Zhang, and Qi Dong. The "visual word form area" is involved in successful memory encoding of both words and faces. Neuroimage, 52 (1):371-378, 2010. 8
Ben Mildenhall, Pratul P Srinivasan, Matthew Tancik, Jonathan T Barron, Ravi Ramamoorthi, and Ren Ng. Nerf: Representing scenes as neural radiance fields for view synthesis. Communications of the ACM, 65(1):99-106, 2021. 19
Kevin Musgrave, Serge Belongie, and Ser-Nam Lim. A metric learning reality check. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXV 16, pp. 681-699. Springer, 2020. 3
Katsuki Nakamura, R Kawashima, Nobuya Sato, A Nakamura, Motoaki Sugiura, T Kato, Kentaro Hatano, K Ito, H Fukuda, T Schormann, al. Funetional delineation of the human occipito- temporal areas related to face and scene processing: a pet study. Brain, 123(:); 2000.
Thomas Naselaris, Kendrick N Kay, Shinji Nishimoto, and Jack L. Gallant. Encoding and decoding in fmri. Neuroimage, 56(2):400-40,, 2011. 2
Kathleen M O'Craven and Nancy Kanwisher. Mental imagery of faces and places activates corre- sponding stimulus-specific brain regions. Journal of cognitive neuroscience, 1216)11013,1023, 2000. 2
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafranicc, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07/193, 202.3 3
Furkan Ozcelik and Rufin VanRullen. Brain-diffuser: Natural scene reconstruction from fmri signals using generative latent diffusion. arXiv preprint arXiv:2303.0533: 2023.
Deepak Pathak, Philipp Krahenbuhl, Jeff Donahue, Trevor Darrell, and Alexei A Efros. Context encoders: Feature learning by inpainting. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 2536-2544, 2016. 3
Michael Peer, Yorai Ron, Rotem Monsa, and Shahar Arzy. Processing of different spatial scales in the human brain. elife, 8:e47492, 2019.
Ian ML Pennock, Chris Racey, Emily J Allen, Yihan Wu, Thomas Naselaris, Kendrick N Kay, Anna Franklin, and Jenny M Bosten. selective. Current Biology, 33(1):134-146, 2023. 2,
regions in the ventral visual pathway are food
Pawet A Pierzchlewicz, Konstantin F Willeke, Arne F Nix. Pavithra Elumalai, Kelli Restivo, Tori Shinn, Cate Nealley, Gabrielle Rodriguez, Saumil Patel, Katrin Franke, et al. Energy guided diffusion for generating neurally exciting images, 2023. 2
14

---

Published as a conference paper at ICLR 2025
Carlos R Ponce, Will Xiao, Peter F Schade, Till S Hartmann, Gabriel Kreiman, and Margaret Livingstone. Evolving images for visual neurons using a deep generative network reveals coding principles and neuronal preferences. Cell, 177(4)999--1099 2019. 3
Sara F Popham, Alexander G Huth, Natalia Y Bilenko, Fatma Deniz, James Gao, Anwar O Nunez- Elizalde, and Jack L Gallant. Visual and linguistic semantic representations are aligned at the border of human visual cortex. Nature neuroscience, 24(1 1):1628-1166, 2021. 2
Jacob Prince, Ian Charest, Jan W Kurzawski, John Pyles, Michael J Tarr, and Kendrick N Kay. Improving the accuracy of single-trial fmri response estimates using glmsingle. eLife, 11:775599, nov 2022. ISSN 2050-084X. doi: 10.7554/eLife.77599 7
Jacob S Prince, George A Alvarez, and Talia Konkle. A contrastive coding account of category selectivity in the ventral visual stream. bioRxiv, pp. 2023-08, 2023. 2
Alec Radford, Jong Wook Kim, Chris Hallacy, A. Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning transferable visual models from natural language supervision. In ICML, 2021. 2,
Maithra Raghu, Thomas Unterthiner, Simon Korbllith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in neural information processing systems, 34:12116-124 11221161111
2021.
Mike Ranzinger, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. Am-radio: Agglomerative vision foundation model reduce all domains into one. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pp. 12490-12500, 2024.
Apurva Ratan Murty, Pouya Bashivan, Alex Abate, James DiCarlo, and Nancy Kanwisher. Com- putational models of category-selective brain regions enable high-throughput tests of selectivity. Nature communications, 12(1):5540, 2021. 2, 3
Ziqi Ren, Jie Li, Xuetong Xue, Xin Li, Fan Yang, Zhicheng Jiao, and Xinbo Gao. Reconstructing seen image from brain activity by visually-guided cognitive representation and adversarial learning. Neurolmage, 228: 17602, 2021. 3
Gabriel Sarch, Michael Tarr, Katerina Fragkiadaki, and Leila Wehbe. Brain dissection: fmri- trained networks reveal spatial selectivity in the processing of natural images. bioRxiv, 2023. doi: 10.1100/0023.0. 29, 542635.. URL https: dectps:/www... wwtpwwwb.rrxi...ooooooo/owww..ww org/ content/eerrlll 2023/11/20
05 29. 5425546665 635.6.5.
Matthew Schultz and Thorsten Joachims. Learning distance metric from relative comparisons. Advances in neural information processing systems, 16, 2003. 2
Paul S Scotti, Mihir Tripathy, Cesar Kadir Torrico Villanueva, Reese Kneeland, Tong Chen, Ashutosh Narang, Charan Santhirasegaran, Jonathan Xu, Thomas Naselaris, Kenneth A Norman, al. Mindeye2: Shared-subject models enable fmri-to-image with hour of data. arXiv preprint 2rXiv:2403.11200 2024. 2.
Katja Sceliger, Umut Güçlü, Luca Ambrogioni, Yagmur Güęlütürk, and Marcel AJ van Gerven. Generative adversarial networks for reconstructing natural images from brain activity. Neurolmage, 181:775-785, 2018. 3
Justine Sergent, Shinsuke Ohta, and Brennan Macdonald. Functional neuroanatomy of face and object processing: positron emission tomography study. Brain, 1 115(1):5-36, 1992. 1, 2
Guohua Shen, Tomoyasu Horikawa, Kei Majima, and Yukiyasu Kamitani. Deep image reconstruction from human brain activity. PLoS computational biology, 15(1)=e1006633, 2019. 3
Ken Shirakawa, Yoshihiro Nagano, Misato Tanaka, Shuntaro C Aoki, Kei Majima, Yusuke Mu- raki, and Yukiyasu Kamitani. Spurious reconstruction from brain activity. arXiv preprint arXiv:2405. 10078, 2024. 10
Eero P Simoncelli and Bruno A Olshausen. Natural image statistics and neural representation. Anmual review of neuroscience, 24(1):1 193-1166,, 2001.
15

---

Published as a conference paper at ICLR 2025
Kihyuk Sohn. Improved deep metric learning with multi-class n-pair loss objective. Advances in neural information processing systems, 29, 2016. 2
A. Stigliani, K. S. Weiner, and K. Grill-Spector. Temporal processing capacity in high-level visual cortex is domain specific. Journal of Neuroscience, 35: 12412-124244 2015, ISSN 0270-6474. doi: 10.1523/INEUROSII..0 4822 482.14..1......15.....
David G Stork and Hugh R Wilson. Do gabor functions provide appropriate descriptions of visual cortical receptive fields? JOSA A, 7(8):1322-1373, 1990.
Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. bioRxiv, pp. 2022-11, 2022. 2,
Antonio Torralba and Aude Oliva. Statistics of natural image categories. Network: computation in neural systems, 14(3):391, 2003. 1
Edgar Y Walker, Fabian H Sinz, Erick Cobos, Taliah Muhammad, Emmanouil Froudarakis, Paul G Fahey, Alexander S Ecker, Jacob Reimer, Xaq Pitkow, and Andreas Tolias. Inception loops discover what excites neurons most using deep predictive models. Nature neuroscience, 22(12): 2060-2065, 2019.
Aria Yuan Wang, Kendrick Kay, Thomas Naselaris, Michael J Tarr, and Leila Wehbe. Better models of human high-level visual cortex emerge from natural language supervision with large and diverse dataset. Nat Mach Intell, 5:1415-1426, 2023a. 2, 3, 4, 7, 22, 23, 24, 25
Feng Wang, Jieru Mei, and Alan Yuille. Sclip: Rethinking self-attention for dense vision-language inference. arXiv preprint arXiv:2312.01597 2023b. 4, 19
Haiguang Wen, Junxing Shi, Wei Chen, and Zhongming Liu, Deep residual network predicts cortical representation and organization of visual features for rapid categorization. Scientific reports, 8(1): 3752, 2018. 2
Zhirong Wu, Yuanjun Xiong, Stella X Yu, and Dahua Lin. Unsupervised feature learning via non- parametric instance discrimination. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3733-3742, 2018. 2
Erie Xing, Michael Jordan, Stuart Russell, and Andrew Ng. Distance metric learning with application to clustering with side-information. Advances in neural information processing systems, 15, 2002.
Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James J DiCarlo. cerformance-optimiize hicrarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, 1 (1(3))88619 8624, 2014. 2
Huzheng Yang, James Gee, and Jianbo Shi. Alignedcut: Visual concepts discovery on brain-guided universal feature space. arXiv preprint arXiv:2406. 18344, 2024a. 2
Huzheng Yang, James Gee, and Jianbo Shi. Brain decodes deep nets. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 23030-23040, 2024b.
Jacob Yeung, Andrew F Luo, Gabriel Sarch, Margaret M Henderson, Deva Ramanan, and Michael J Tarr. Neural representations of dynamic visual stimuli. arXiv preprini 2rXiv:2406.02659, 2024. 2
Xiaohua Zhai, Basil Mustafa, Alexander Kolesnikov, and Lucas Beyer. Sigmoid loss for language image pre-training. In Proceedings of the IEEE/CVE International Conference Computer Vision, pp. 11975-11986, 2023. 2, 3,7
Yucheng Zhao, Guangting Wang, Chong Luo, Wenjun Zeng, and Zheng-Jun Zha. Self-supervised visual representations learning by contrastive mask prediction. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 10160-10169, 2021.
Chong Zhou, Chen Change Loy, and Bo Dai. Extract free dense labels from clip. In European Conference on Computer Vision, pp. 696-712. Springer, 2022. 4, 19
Jinghao Zhou, Chen Wei, Huiyu Wang, Wei Shen, Cihang Xie, Alan Yuille, and Tao Kong. ibot: Image bert pre-training with online tokenizer. arXiv preprint arXiv:21/1.07832 2021.
16

---

Published as a conference paper at ICLR 2025
A
APPENDIX
Sections
1. Implementation and fMRI processing details (A.1)
2. Comparing CLIP granularity and brain saliency (A.2)
3. Additional encoder test set R for CLIP, DINO, and SigLIP (A.3)
4. Visualization of ground truth functional localizer statistics (A.4)
5. Visualization of encoder weight UMAP for all subjects using CLIP backbone (A.5)
6. Visualization of encoder weight UMAP for all subjects using DINO backbone (A.6)
7. Visualization of encoder weight UMAP for all subjects using SigLIP backbone (A.7)
8. Additional visualization of fMRI grounding using CLIP backbone encoder (A.8)
9. Additional visualization of fMRI grounding using DINO backbone encoder (A.9)
10. Additional visualization of fMRI grounding using SigLIP backbone encoder (A.10)
17

---

Published as a conference paper at ICLR 2025
A.1 IMPLEMENTATION AND FMRI PROCESSING DETAILS
Encoder training. Our experiments utilize mixture of GPUs including Nvidia V100 (16GB and 32GB), 2080 Ti, A6000, 6000Ada, L40S, and 4090 cards. The network training code is implemented in PyTorch. For encoder training, we employ the Adam optimizer with decoupled weight decay of x 10-2, The initial learning rate is set to 10-4 and decays exponentially to 1.5 10-1 over 100 epochs. Each subject is trained independently, All backbone networks are frozen, and operate in fp16 mode.
For encoder training, we always utilize the network's native input resolution. For CLIP, we resize images to 224 x 224. For DINO, we resize images to 518 x 518. For SigLIP using the AM-RADIO backbone, and resize images to 768 x 768. After resizing, pixel-wise value scaling between 0.95 and 1.05, followed by normalization using each network's respective image mean and variance. Before network input, images are randomly offset by up pixels horizontally and vertically, with edge padding filling the resulting empty pixels. Independent Gaussian noise (μ 0, o² 0.05) added each pixel.
augment the images with random
Feature extraction. We perform positional encoding interpolation for CLIP and DINO networks in order to extract higher resolution embeddings. For CLIP ViT-B B/ 6 we modify the network to accept images of 896 896 (4x upsampling: 56 56 final patch resolution); For DINOV2 WiT-B/14+res we modify the network to accept images 1036 x 1036 (2x upsampling: 74 x 74 final patch resolution); For SigLIP based Nvidia's RADIOv2.. 5 ViT-B/15 6, we do not perform upsampling, and have 48 x 48 final patch resolution. As noted in the AM-RADIO paper, the spatial patch features of these networks are highly robust to positional encoding upsampling. We perform 51 augmentation steps for CLIP, where the first augmentation step is null (no shift or flipping). For DINO and SigLIP, as these networks contain registers which mitigate the artifacts to certain degree, we perform 25 augmentation steps.
For CLIP. we utilize the NACLIP self attention modification, with gaussian std set to 10.0. For DINO and SigLIP, we use the MaskCLIP self-attention modification. For SigLIP specifically, since we are using the AM-RADIO implementation, we apply their provided SigLIP adapter head to the extracted features. As shown in section A.7, this enables zero-shot probing with the official Sigl .IP text encoder using the extracted dense features.
Computational cost. Our "Learning- Distillation Module" is highly efficient. With pre- extracted dense features for each augmentation, we observe our procedure generally takes less than 0.2 seconds on a GPU, this roughly 100x faster than "Denoising Vision Transformers" on the same hardware and similarly ignoring the feature extraction cost.
CLIP sentences. We define a set of natural language captions to help us evaluate the alignment between fMRI region-wise relevance maps and concepts. For every caption and every image, we compute a relevance map using the CLIP text encoder. The category that contains the caption with the highest pearson correlation to the fMRI relevance map is assigned as the category.
face_clas ass ["A face facing the camera", "A photo of a face", "A photo of human face", "A photo of faces", "A photo of person's face", "A person looking at the camera", "People looking at the camera", "A portrait of person", "A portrait photo"]
class ["A photo of torso", "A photo of torsos", "A photo of limbs", "A photo of bodies", "A photo of person", "A photo of people", "A photo of body", "A person's arms", "A person's legs", "A photo of hands"]
scene_clase = ['A photo of bedroom", "A photo of an office", "A photo of a hallway", "A photo of a doorway", "A photo of interior design", "A photo of a building", "A photo of a house", "A photo of nature", "A photo of landscape", "A landscape photo", "A photo of trees", "A photo of grass"]
food class ["A photo of food", "A photo of cuisine", "A photo of fruit", "A photo of foodstuffa", "A photo of a meal", "A photo of bread", "A photo of rice", "A photo of snack", "A photo of
18

---

Published as a conference paper at ICLR 2025
pastries", "A photo of vegetables", "A photo of pizza", "A photo of soup", "A photo of meat", "A photo of candy"]
text_class = ["A photo of words", "A photo of glyphs", "A photo of glyph", "A photo of text", "A photo of numbers", "A photo of a letter", "A photo of letters", "A photo of writing", "A photo of text on an object"]
IMRI data processing. All of our analysis is done in subject native space voxels in 1.8mm² resolution (func1pt and we use the beta values from betas wutthff.GMMdenee Cortical voxels selected based the nsdgeneral mask.
In order to select visually responsive voxels for Figure 3, we utilize the HCP parcellation which yields 180 regions. For each region in each subject, we compute the average noise ceiling using NSD provided data. We rank the regions within each subject, and average the ranks across all subjects. The best 25% (45 out of 180) regions by noise ceiling selected. We further mask out voxels which labeled as early visual (VI~ V4).
Intuition for learning-fret distillation. We are specifically motivated by literature in learning dense semantic embeddings. In particular, Kobayashi al. (2022) proposed learning a dense 3D semantic representation. However Kobayashi et al. (2022) uses LSeg (Li et al., 2022) - supervised open- veee-oocay segmentation network which demonstrates relatively poor generalization to categories outside of the training set (Kerr et al., 2023). As we do not wish to restrict the hypothesis space relative cortical selectivity, we motivated explore methods which preserve the original representational capability of vision foundation models. By default, the dense token embeddings do lie in the same semantic space as the summary token. However modifications like MaskCLIP (Zhou et al., 2022), SCLIP (Wang et al., 2023b), and NACLIP (Hajimiri et al., 2024) propose changes to the self-attention mechanism that ensure the dense embeddings are in the same space as the summary token. We utilize NACLIP for CLIP (or models without registers). As register tokens have not been explored in the context of self-attention modifications, we use MaskCL.P for models with registers. Without further distillation, these methods yield dense embeddings that have artifacts.
Recent work has proposed to learn consistent 3D representations using NeRF and feature fields (Mildenhall et al., 2021; Kobayashi et al., 2022). These methods demonstrate that a spa- tially consistent representation can be learned in the presence of noise (pixel noise or measurement noise) by utilizing photometric 3D consistency framework with multiple views. As we do not have multiple views of single image in 2D, augment images with horizontal flips and verti- cal/horizontal offsets to generate synthetic views. These augmentations do not change the semantics of scene. We observe that because the artifacts not static relative scene content, augmentations can effectively render artifacts into "measurement noise".
Originally NeRF and feature fields perform very expensive neural field learning with gradients and MSE loss. Faster learning and inference procedures are required to facilitate data-driven studies of the visual cortex using thousands of images. We show that averaging over views achieves the same mathematical result as MSE driven optimization.
19

---

Published as a conference paper at ICLR 2025
A.2
COMPARING CLIP GRANULARITY AND BRAIN SALIENCY
RGB
Face
Legs
Arms
Street
CLIP text
[MRI region
Distilled feats
Faces
Bodies
Places
Hands holding phone
RGB
Faces
Wood table
Food
CLIP text
FMRI region
Distilled feats
Faces
Bodies
Places
Food
Figure S.I: Comparing the granularity of CLIP embeddings with brain region saliency. On the left, we visualize the image and the UMAP of the distilled NACLIP embeddings. On the top right, we probe the dense features using the CLIP-text encoder. On the bottom right, we visualize the relevance map using the fMRI encoder weights for t > 2 functional regions from S1. In both cases, the probing done with a cosine similarity. We find that brain data usually has semantically coarser attributions. Likely due to NSD data co-occurrence statistics, we find that neural data does not consistently separate faces bodics, while CLIP itself can have very fine mne-nraieed representations.
20

---

Published as a conference paper at ICLR 2025
RGB
Sunglasses
Food
Sea
Wood table
CLIP text
[MRI region
Distilled feats
Faces
Bodies
Places
Food
RGB
Faces
Sea
Food
CLIP text
fMRI region
Distilled feats
Faces
Bodies
Places
Food
Figure S.2: Comparing the granularity of CLIP embeddings with brain region saliency. On the left, visualize the image and the UMAP of the distilled NACLIP embeddings. On the top right, we probe the dense features using the CLIP-text encoder. On the bottom right, we visualize the relevance map using the fMRT encoder weights for i. 2 functional regions from S1. In both cases, the probing is done with a cosine similarity. We find that brain data usually has semantically coarser attributions. Likely due NSD data co-occurrence statistics, consistently separate faces & bodies, and attributes food selectivity to small objects or high saturation image regions (note sunglasses in top image, and pink dress of bottom image) food itself is often both small and high saturation. Our method can reveal surprising behavior of fMRI encoders.
find that neural data does
21

---

Published as a conference paper at ICLR 2025
A.3 ADDITIONAL ENCODER TEST SET R FOR CLIP, DINO, AND SIGLIP
Subjects 1, 2, are those that completed the full scan of 10, 000 images where each image was viewed 3 times. They are also the subjects reported to have the highest noise ceiling in NSD (higher is better).
(a)
S1
CLIP
DINO
SigLIP
S2
0.4
x=y
(b)
0.6
DINO R2
DINO R2
S1
SigLIP
0.4
CLIP R²
SigLIP R
CLIP R²
0.
DINO R
DINO R2
SigLIP
0.4
S2
00
CLIP R²
SigLIP
CLIP
Figure S.3: Test set R² using different encoder backbones on subjects S1 & S2. (a) We validate each encoder R2 test and find that all three models achieve very high performance (comparable to Wang et al. (2023a)). (b) The voxel-wise correlation test set R2 for the three models. CLIP and SigLIP, which rely language supervision, achieve higher performance than DINO (which trained via self-supervision with images).
22

---

Published as a conference paper at ICLR 2025
(a)
S3
CLIP
DINO
SigLIP
S4
x=y
(b)
28
DINO R2
DINO R2
S3
SiLLIP
022
68
0.0
04
05
84
CLIP R2
SigLIP
CLIP R2
0.
DINO R2
DINO R2
S4
SiGLIP
0.4
90
CLIP R2
SigLIP R2
CLIP R?
Figure S.4: Test set R² using different encoder backbones on subjects S3 & S4. (a) We validate each encoder R on test set and find that all three models achieve very high performance (comparable Wang et al. (2023a)). (b) The voxel-wise correlation test set Rd for the three models. CLIP and SigLIP, which rely on language supervision, achieve higher performance than DINO (which trained via self-supervision with images).
23

---

Published as a conference paper at ICLR 2025
(a)
S5
CLIP
DINO
SigLIP
S6
x=y
(b)
DINO R²
DINO R2
S5
SigLIP
CLIP R
SigLIP R2
CLIP R?
0.51
SigLIP R2
DINO R2
DINO R2
S6
0.5
0.2
0.4
86
08
10C
0.2
0.4
66 0.8
CLIP
SigLIP R²
CLIP R2
Figure S.5: Test set R² using different encoder backbones on subjects $5 & S6. We validate each encoder R on test set and find that all three models achieve very high performance (comparable Wang et al. (2023a)). (b) The voxel-wise correlation test set Rd for the three models. CLIP and SigLIP, which rely on language supervision, achieve higher performance than DINO (which trained via self-supervision with images).
24

---

Published as a conference paper at ICLR 2025
(a)
S7
CLIP
DINO
SigLIP
S8
x=y
(b)
SigUP R
DINO R2
DINO R2
S7
0.4
CLIP R
SigLiP
CLIP R2
0.51
SigLIP R2
DINO R2
DINO R2
S8
0.5
0.2
0.4
86 08
CLIP R²
SigLIP R
CLIP R²
Figure S.6: Test set R² using different encoder backbones on subjects S7 & S8. (a) We validate each encoder R on test set and find that all three models achieve very high performance (comparable Wang et al. (2023a)). (b) The voxel-wise correlation test set Rd for the three models. CLIP and SigLIP, which rely on language supervision, achieve higher performance than DINO (which trained via self-supervision with images).
25

---

Published as a conference paper at ICLR 2025
A.4 VISUALIZATION OF GROUND TRUTH FUNCTIONAL LOCALIZER STATISTICS
In this section, we show the ground truth functional localizer I-statistic provided by NSD. As NSD does not provide food localizer, we obtain the food mask from Jain et al. (2023). During the UMAP, we do not provide our method with the ground truth functional localizer information, and these t-statistics are provided here for reference only.
Faces
Places
RSC
OPA
OPA
EBA
aTL.
FAFFA4
FFA
FEA-2
faoma
PPA
PPA
Bodies
Words
t-value
-8
8
Figure S.7: Sematic category functional localizer statistics for S1. We visualize the ground truth functional localizer i-statistic provided by NSD. Higher indicates stronger selectivity semantic category.
Faces
Places
Bodies
Words
t-value
-8
8
Figure S.8: Sematic category functional localizer statisties for S2. We visualize the ground truth
functional localizer i-statistic provided by NSD. Higher indicates stronger selectivity to a semantic category.
26

---

Published as a conference paper at ICLR 2025
Faces
Places
Bodies
Words
t-value
-8
8
Figure S.9: Sematic category functional localizer statisties for SS. visualize the ground truth functional localizer i-statistic provided by NSD. Higher indicates stronger selectivity semantic category.
Faces
Places
Bodies
Words
t-value
-8
8
Figure S.10: Sematic category functional localizer statisties for S7. We visualize the ground truth functional localizer i-statistic provided by NSD. Higher indicates stronger selectivity to semantic category.
27

---

Published as a conference paper at ICLR 2025
A.5 VISUALIZATION OF ENCODER WEIGHT UMAP FOR ALL SUBJECTS USING CLIP BACKBONE
We utilize OpenAT's official Vi T-B/16 the encoder backbone, and visualize the UMAP as applied to the fMRI encoder weights and dense features.
28

---

Published as a conference paper at ICLR 2025
Woman
Food
Furniture
Opera House
UMAP3
GIAA
UMAP2
9BA
Food
Food
OTE
PPA
S1
FTA
$2
Superior
Superior
Anterior
Anterior
Left Hemisphere
Right Hemisphere
Figure S.11: Encoder with CLIP backbone UMAP transform results for subjects S1-S4. We normalize all encoder weights to unit norm prior to UMAP, and use an angular metric for the fitting and projection stage. The UMAP is fitted on S1 and reused across all subjects and images. Both patch and voxel vectors are projected onto the space of natural images prior to transform using softmax weighted sum similar BUAiA For each quartet of images, the content is follows Top left: Original RGB image; Top right: Dimension reduction of BrainSAIL. embeddings for the image using CLIP features; Bottom: Two text queries using CLIP text branch showing language-indicate relevance results.
29

---

Published as a conference paper at ICLR 2025
Woman
Food
Furniture
Opera
UMAP3
PSC
UMAP2
Fond
FA all-
focas
S5
S6
Superior
Superior
Anterior
Anterior
Left Hemisphere
S8
Right Hemisphere
Figure S.12: Encoder with CLIP backbone UMAP transform results for subjects S5-S8. We normalize all encoder weights to unit norm prior to UMAP, and use an angular metric for the fitting and projection stage. The UMAP is fitted on S1 and reused across all subjects and images. Both patch and voxel vectors are projected onto the space of natural images prior to transform using softmax weighted sum similar BUAiA For each quartet of images, the content is follows Top left: Original RGB image; Top right: Dimension reduction of BrainSAIL. embeddings for the image using CLIP features; Bottom: Two text queries using CLIP text branch showing language-indicate relevance results.
30

---

Published as a conference paper at ICLR 2025
A.6 VISUALIZATION ENCODER WEIGHT UMAP FOR ALL SUBJECTS USING DINO BACKBONE
We utilize Meta's official DINOV2 ViT-B/1 3/14+reg the encoder backbone, and visualize the UMAP as applied to the fMRI encoder weights and dense features. The UMAP method nonpara- metric. and the output can depend on the random seed. Similar colors for UMAP applied to DINO encoder weights does not indicate any specific similarity to CLIP results in the prior section.
As DINO does have encoder, cannot visualize text-based localization probes.
31

---

Published as a conference paper at ICLR 2025
UMAP3
RSC
OPA
UMAP2
UMAAP
EBA
Feston
legion
Faces
S1
$2
$3
Superior
Superior
Anterior
Left Hemisphere
S4
Right Hemisphere
Figure 1.: Encoder with DINO backbone UMAP transform results for subjects S1-S4. We normalize all encoder weights to unit norm prior to UMAP, and use angular metric for the fitting and projection stage. The UMAP is fitted on and reused across all subjects and images. Both patch and voxel vectors projected onto the space of natural images prior to transform using softmax weighted sum similar to BrainSCUBA. For each pair of images, the content is as follows Left: Original RGB image; Right: Dimension reduction of BrainSAIL embeddings for the image using DINO features.
32

---

Published as a conference paper at ICLR 2025
UMAP3
UMAP2
UMAAP
EBA
region
SIL FAA 2
1 STL
$5
S6
S7
Superior
Superior
Anterior
Left Hemisphere
S8
Right Hemisphere
Figure S.14: Encoder with DINO backbone UMAP transform results for subjects S5-S8. We normalize all encoder weights to unit norm prior to UMAP, and use angular metric for the fitting and projection stage. The UMAP is fitted on and reused across all subjects and images. Both patch and voxel vectors projected onto the space of natural images prior to transform using softmax weighted sum similar to BrainSCUBA. For each pair of images, the content is as follows Left: Original RGB image; Right: Dimension reduction of BrainSAIL embeddings for the image using DINO features.
33

---

Published as a conference paper at ICLR 2025
A.7 VISUALIZATION OF ENCODER WEIGHT UMAP FOR ALL SUBJECTS USING SIGLIP BACKBONE
We utilize Nvidia's RADIOv2. 5 ViT-B/16 6 implementation for SigLIP. Nvidia's implementation was choosen as the original Google implementation did not utilize a ViT with a attention [CLS] token. We visualize the UMAP applied the fMRI encoder weights and dense features. The UMAP method is nonparametric, and the output can depend on the random seed. Similar colors for UMAP applied to SigLIP encoder weights does not indicate any specific similarity to DINO or CLIP results in the prior section.
34

---

Published as a conference paper at ICLR 2025
Woman
Food
Food
Furniture
Opera House
UMAP3
GFA
UMAP2
EBA
Food
Food
OTL
PAA
S1
S2
Superior
Superior
Anterior
Anterior
Left Hemisphere
Right Hemisphere
Figure S.15: Encoder with SigLIP backbone UMAP transform results for subjects $1-S4. We normalize all encoder weights to unit norm prior to UMAP, and use an angular metric for the fitting and projection stage. The UMAP is fitted on S1 and reused across all subjects and images. Both patch and voxel vectors are projected onto the space of natural images prior to transform using softmax weighted sum similar BrainSCUBA. For each quartet of images, the content is follows Top left: Original RGB image; Top right: Dimension reduction of BrainSAIL. embeddings for the image using SigLIP features; Bottom: Two text queries using SigLIP text branch showing language-indicate relevance results.
35

---

Published as a conference paper at ICLR 2025
Woman
Food
Food
Furniture
Opera
UMAP3
RSC
OORA
UMAP2
FBA
EBA
Food
FF
FAAZ
RBA
fsoes
$5
S6
Superior
Superior
Anterior
Anterior
Left Hemisphere
S8
Right Hemisphere
Figure S.16: Encoder with SigLIP backbone UMAP transform results for subjects $5-S8. We normalize all encoder weights to unit norm prior to UMAP, and use an angular metric for the fitting and projection stage. The UMAP is fitted on S1 and reused across all subjects and images. Both patch and voxel vectors are projected onto the space of natural images prior to transform using softmax weighted sum similar BrainSCUBA. For each quartet of images, the content is follows Top left: Original RGB image; Top right: Dimension reduction of BrainSAIL. embeddings for the image using SigLIP features; Bottom: Two text queries using SigLIP text branch showing language-indicate relevance results.
36

---

Published as a conference paper at ICLR 2025
A.8 ADDITIONAL VISUALIZATION OF GROUNDING USING CLIP BACKBONE
37

---

Published as a conference paper at ICLR 2025
Face Voxels
PCA (feehures)
Grownsing [BrairSnit]
Place Voxels
ASD (ININI)
PCA (feehuees)
Grounding (trainssil
Word Voxels
NSD (fsle))
PCA (features)
Grounding nn (BreinSA
Body Voxels
NSO (fall))
PCA (featires)
Grounding (rrainsii
Food Voxels
INSO (TMAN))
PCA (DramiSei) (ffestune))
grounding
Figure S.17: Grounding results using BrainSAIL. for S1 CLIP backbone.. We visualize the top test set images according to NSD fMRI for each category selective region. For each image, we also visualize the image-wise PCA for the distilled dense features. Note the PCA basis here is computed imagewise. For each image, we further visualize the feature relevancy map for the category selective voxels illustrating that this method extracts the semantically relevant regions in complex compositional images.
38

---

Published as a conference paper at ICLR 2025
Face Voxels
PCA (feetures)
Grownding (BrainSnll)
Place Voxels
ASD (NNIN
PCA (feehueee)
Grounding (BraissAl
Word Voxels
NSD (fslog)
PCA (features)
Grounding nng (BriinSA
Body Voxels
NSO (fall))
PCA (features
Grounding (trainSiit
Food Voxels
INDD
PCA (DramSSAi)) (ffesuune))
Grounding
Figure S.18: Grounding results using BrainSAIL. for S2 CLIP backbone.. We visualize the top test set images according to NSD fMRI for each category selective region. For each image, we also visualize the image-wise PCA for the distilled dense features. Note the PCA basis here computed imagewise. For each image, we further visualize the feature relevancy map for the category selective voxels illustrating that this method extracts the semantically relevant regions in complex compositional images.
39

---

Published as a conference paper at ICLR 2025
Face Voxels
PCA (feetures)
Grownding [BrainSnit)
Place Voxels
ASD (ININ
PCA (feeturee)
Grounding (Braissilt
Word Voxels
NSD (1M66)
PCA (features)
Grownd (BriinSA
Body Voxels
NSO (f6ll))
PCA (featires)
Grounding (trainsii
Food Voxels
INDD (TMIN))
Grounding PCA (BramSSni)) (ffetuunes)
Figure S.19: Grounding results using BrainSAIL. for S5 CLIP backbone.. We visualize the top test set images according to NSD fMRI for each category selective region. For each image, we also visualize the image-wise PCA for the distilled dense features. Note the PCA basis here computed imagewise. For each image, we further visualize the feature relevancy map for the category selective voxels illustrating that this method extracts the semantically relevant regions in complex compositional images.
40

---

Published as a conference paper at ICLR 2025
Face Voxels
PCA (feeturee)
Grownding (BrainSAl)
Place Voxels
ASD (NNAR
PCA (feehuree)
Grounding (BrainsAl
Word Voxels
NSS (Mi66))
PCA (feabures)
Grounding (BreinSA
Body Voxels
NSO (f6ll))
PCA (feat (featines)
Grounding (rrainsii
Food Voxels
INSD (Thin))
Grounding PCA (framiSni) (ffetuune))
Figure S.20: Grounding results using BrainSAIL. for S7 CLIP backbone.. We visualize the top test set images according to NSD fMRI for each category selective region. For each image, we also visualize the image-wise PCA for the distilled dense features. Note the PCA basis here computed imagewise. For each image, we further visualize the feature relevancy map for the category selective voxels illustrating that this method extracts the semantically relevant regions in complex compositional images.
41

---

Published as a conference paper at ICLR 2025
A.9 ADDITIONAL VISUALIZATION OF GROUNDING USING DINO BACKBONE
42

---

Published as a conference paper at ICLR 2025
Face Voxels
PCA (feehires)
Grownding [BrainSnlt)
Place Voxels
NND (ININ
PCA (feehree))
Groundsil (breinsnlt)
Word Voxels
NSD (fsle))
PCA (features)
Grounding nn (BreinSA
Body Voxels
NSO (fall))
PCA (featunes)
Grounding Urrinsiil
Food Voxels
IN50 (TMAN))
PCA (DramISAi) (ffesuune))
grounding
Figure S.21: Grounding results using BrainSAIL. for S1 DINO backbone.. We visualize the top test set images according to NSD fMRI for each category selective region. For each image, we also visualize the image-wise PCA for the distilled dense features. Note the PCA basis here is computed imagewise. For each image, we further visualize the feature relevancy map for the category selective voxels illustrating that this method extracts the semantically relevant regions in complex compositional images.
43

---

Published as a conference paper at ICLR 2025
Face Voxels
PCA (feetures)
Grownding [BrainSAil)
Place Voxels
ASD (NNIN
PCA (feehueee)
Grounding (Brainssll
Word Voxels
NSD (fslog)
PCA (features)
Grownding ngg (BrainSA
Body Voxels
NSO (fall))
PCA (feat
Grounding (irainSiiy
Food Voxels
IND (TMIN))
Grounding PCA (DramSSAL) (ffetuune))
Figure S.22: Grounding results using BrainSAIL. for S2 DINO backbone.. We visualize the top test set images according to NSD fMRI for each category selective region. For each image, we also visualize the image-wise PCA for the distilled dense features. Note the PCA basis here computed imagewise. For each image, we further visualize the feature relevancy map for the category selective voxels illustrating that this method extracts the semantically relevant regions in complex compositional images.
44

---

Published as a conference paper at ICLR 2025
Face Voxels
PCA (feetures)
Grownding [BrairSnit)
Place Voxels
ASD (ININ
PCA (feehuees)
Grounding (Brainssll)
Word Voxels
NSD (1M66))
FCA (features)
Grounding ng (frinnSA
Body Voxels
NSO (f6ll))
PCA (featines)
Grounding (Brannsii
Food Voxels
NSD (TMIN))
Grounding PCA (framSSAL)) (ffattune))
Figure S.23: Grounding results using BrainSAIL. for S5 DINO backbone.. We visualize the top test set images according to NSD fMRI for each category selective region. For each image, we also visualize the image-wise PCA for the distilled dense features. Note the PCA basis here computed imagewise. For each image, we further visualize the feature relevancy map for the category selective voxels illustrating that this method extracts the semantically relevant regions in complex compositional images.
45

---

Published as a conference paper at ICLR 2025
Face Voxels
PCA (feehures)
Grownding [BrainSAll)
Place Voxels
ASD (NNA
PCA (feehuree)
Groundkil (Brainsklt)
Word Voxels
N5S (1460)
PCA (eeatures)
Grounding ng (BreinSA
Body Voxels
NSO (f6ll))
PCA (featines)
Grounding (trainsni
Food Voxels
INSD (ThIN))
Grounding PCA (BramISAI) (featueee)
Figure S.24: Grounding results using BrainSAIL. for S7 DINO backbone.. We visualize the top test set images according to NSD fMRI for each category selective region. For each image, we also visualize the image-wise PCA for the distilled dense features. Note the PCA basis here computed imagewise. For each image, we further visualize the feature relevancy map for the category selective voxels illustrating that this method extracts the semantically relevant regions in complex compositional images.
46

---

Published as a conference paper at ICLR 2025
A.10 ADDITIONAL VISUALIZATION OF GROUNDING USING SIGLIP BACKBONE
47

---

Published as a conference paper at ICLR 2025
Face Voxels
PCA (feetures)
Grownding [BrainSnit)
Place Voxels
NSD (ININ
PCA (feehueee)
Grounding (BrainSsit)
Word Voxels
NSD (fsle))
PCA (features)
Groundingg nng (BreinSi
Body Voxels
NSO (fall))
PCA (featires)
Grounding Urransiiiy
Food Voxels
INSD
PCA (DramSSii (efesuune))
grounding
Figure S.25: Grounding results using BrainSAIL. for S1 SigLIP backbone.. We visualize the top test set images according to NSD fMRI for each category selective region. For each image, we also visualize the image-wise PCA for the distilled dense features. Note the PCA basis here computed imagewise. For each image, we further visualize the feature relevancy map for the category selective voxels illustrating that this method extracts the semantically relevant regions in complex compositional images.
48

---

Published as a conference paper at ICLR 2025
Face Voxels
PCA (feetures)
Grownding [BrairSnit)
Place Voxels
ASD (NNI
PCA (feehures)
Grounding (Ir Uirainsnily
Word Voxels
NSD (fslog)
PCA (features)
Grounding ngg (BrainSA
Body Voxels
NS0 (1all))
PCA (feat
Grounding (irainsiiy
Food Voxels
IND
Grounding PCA (framiSAi) (ffetuune))
Figure S.26: Grounding results using BrainSAIL. for S2 SigLIP backbone.. We visualize the top test set images according to NSD fMRI for each category selective region. For each image, we also visualize the image-wise PCA for the distilled dense features. Note the PCA basis here computed imagewise. For each image, we further visualize the feature relevancy map for the category selective voxels illustrating that this method extracts the semantically relevant regions in complex compositional images.
49

---

Published as a conference paper at ICLR 2025
Face Voxels
PCA (feehures)
Grownding [BrainSnit)
Place Voxels
ASD (NNIN
PCA (feehueee)
Grounding (br (brainsiit
Word Voxels
NSD (1M66))
PCA (features)
Grounding ng (BriinSA
Body Voxels
NSO (16ll))
PCA (featines)
Grounding (BrainSiit
Food Voxels
INSD
Grounding PCA (DramiSAi)) (ffetuune))
Figure S.27: Grounding results using BrainSAIL. for S5 SigLIP backbone.. We visualize the top test set images according to NSD fMRI for each category selective region. For each image, we also visualize the image-wise PCA for the distilled dense features. Note the PCA basis here is computed imagewise. For each image, we further visualize the feature relevancy map for the category selective voxels illustrating that this method extracts the semantically relevant regions in complex compositional images.
50

---

Published as a conference paper at ICLR 2025
Face Voxels
PCA (feetrees)
Grownding [BrairSAlt
Place Voxels
ASD (NNA
PCA (feesurre)
Grounding (Brainssit)
Word Voxels
NSD (1N66)
PCA (features)
Grounding ng (BreinSA
Body Voxels
NSO (16ll))
PCA (feat (featines)
Grounding (rrainsti
Food Voxels
INSD (ThiN))
PCA (DramSSii (ffestune))
Grounding
Figure S.28: Grounding results using BrainSAIL. for S7 SigLIP backbone.. We visualize the top test set images according to NSD fMRI for each category selective region. For each image, we also visualize the image-wise PCA for the distilled dense features. Note the PCA basis here computed imagewise. For each image, we further visualize the feature relevancy map for the category selective voxels illustrating that this method extracts the semantically relevant regions in complex compositional images.
51


Reviews:
[{"review_id": "monNLhuk9p", "reviewer": "Reviewer_PZHq", "summary": "The paper proposes a method, BrainSAIL, which aims to disentangle complex images into their semantically meaningful components and locate them to brain voxels or regions. With the prevalence of pretrained models (trained on very large amounts of data), deep learning offers a promising way to explore how semantics are organized in the brain cortex. Compared to prior methods, BrainSAIL focuses on selectivity in single-object images at the broad category level, thereby enabling a richer decomposition grounded in the full semantic complexity of natural visual experiences.", "questions": "Was the same adapter used in Figures 1a and 1b? Also, were the adapter parameters frozen in Figure 1b? If they were frozen, why not use clean dense features to train the adapter?\n\nHow does the quality of the dense features impact the adapter's behavior? I suggest adding an ablation study to explore this effect.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This is clearly written paper and makes a clear contribution. The idea of isolating specific visual features to determine selectivity effects in different cortical areas is novel and interesting. The proposed method can be used to explore the selectivity of higher visual cortex with respect to localized scene structure and image properties. This work achieves promising open vocabulary CLIP-based segmentation results.", "weaknesses": "The current experimental comparisons (open-vocabulary segmentation) are limited, which restricts a comprehensive evaluation of the proposed method. The proposed approach appears to closely resemble BrainSCUBA. Certain technical details are unclear, which makes it challenging to fully understand or reproduce the method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hI1zm9UfaB", "reviewer": "Reviewer_yyM7", "summary": "Methods:\n- Embeddings from deep neural networks are used to predict fMRI recorded brain-responses.\n- Features are extracted from the model and input to a denoising process. The smoothed features are then input to the fMRI encoder (I think) to create spatial attribution maps.\n- UMAP is applied to the encoder parameters, which uncovers previously reported category selective regions for faces, places, words, food, and bodies.\n- The spatial attribution maps are correlated with pixel depth, color saturation, and color luminance to create voxel-wise correlation values for these image properties. Place areas were found to be more correlated with depth, food areas with color, and OPA with color/luminance.", "questions": "1. My understanding is that the voxel-wise adapters predict a scalar brain response from an M-dimensional output of the backbone model. How is the voxel-wise adapter then applied to the dense features? I am thinking the dense features have a shape (H, W, M), and the adapter transforms the last dimension to get a scalar (H, W) activation map (figure 1.b). I found this a bit unclear in the paper.\n\n2. The learning free distillation is applying random spatial transformations to the input image, applies the inverse transformation to the features, and then averages them all together. Is my understanding correct? \n\n3. I do not understand what equations 3-5 are about.\n\n4. Figure 4 states that the UMAP basis is computed image-wise. I found it unclear how this was done, and what is the significance of an image-wise UMAP?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper is well written and clear. The main contribution is the use of model features to create voxel and image-wise spatial contribution maps. This is quite useful as an interpretation technique. The use of pixel-wise metrics like depth, saturation, and luminance are a powerful extension of the method.", "weaknesses": "All of the investigated category-selective areas (food, places, words, faces, bodies) are previously reported. Would be interesting so see this tested on less common or more fine-grained categories.\n\nI found the explanation of the methods is a bit confusing (see questions). Could use clearer high-level descriptions before diving into the finer details.\n\nI think there should be a visual comparison of attribution maps for different voxel categories with the same images. I was able to compare a few in figure 3 since there are some repeated images in words/food and face/body voxels. If the method is working then the word voxels should highlight words more than voxels for other categories. I would expect to see the face voxels highlighting faces a lot more than body parts, and the opposite for body voxels. However in the picture of the baby shared between the body/face voxels I don't see much of a difference in their attribution maps.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "qMumcqKaMM", "reviewer": "Reviewer_PeSm", "summary": "This paper aims to explore the semantic topography in human visual cortex using fMRI. The main goal is to disambiguate the typically confounded nature of multiple categories which is a feature/issue in naturalistic images. The trend towards naturalistic datasets in recent years necessitates more advanced methods to be able to deal with such confounds and this work takes a good step forward in the right direction. This paper then goes on to look at what regions of human visual cortex are driven by low-level image features, providing a complementary view into how the authors’ method can be used beyond studying categorical selectivity. This method provides a good solution to explore such potential confounding effects in determining what is driven by high or low level visual information.\n\nIt was a very nice paper to read and shows the authors' clear expertise and attention to detail in terms of the questions we should be asking in the community.", "questions": "- “We validate this dense feature mapping method on a large-scale fMRI dataset consisting of human participants viewing many thousands of diverse natural images that span a wide range of semantic categories and visual statistics (Allen et al.,2022).”\n    - The work by Kamitani recently showed that UMAP gave something like 40 clusters of semantic diversity at a broad level, forcing us to re-evaluate our assumed understanding of just how semantically varied NSD actually is. Do you believe the findings in the \"spurious reconstructions\" paper have an implication on the notion of semantic diversity as described in your work?\n\n- The introduction has repeated references to BrainSAIL’s dense embedding framework but I’m not sure why this point is continually brought up as the vision model embeddings on the natural image are dense to begin with and I can’t see that the implementation of a step to make embeddings more dense is applied anywhere, so I was a bit lost on this point. You have the adapters and a linear probe and everything always seems to be in a dense (not non-sparse) setting. Some clarification on this point would be welcome in the paper as you seem to mean something different to how I understand the same phrasing.\n\n- Was there supposed to be a section explaining the data choices for the brain data (surface vs voxels, subject-space vs template-space) etc.? This is kind of left to be assumed from other sections of the data but I think a small section in the supplemental outlining the data usage choices more specifically would render the procedure more complete with regard to necessary experimental details. For example, the fact you’re using the masks in Jain et al. (2023) implies working in standard (normalised) template space, but I sort of expected being able to verify this explicitly via some technical description somewhere in the supplemental materials.\n\n- You described how you selected varying functional regions from the dataset (food masks in Jain et al. (2023) and t > 2 in the Stigliani fLoc dataset) but the selected voxels being plotted, the selection of them, is not described and also a piece of missing information. Was this taken by looking at noise ceiling values calculated in NSD and if so, what value was used as the threshold?\n- Lines 427-429: you identify the areas surrounding the FFA to be selective for colour, and point out this corresponds to the food area previously identified by others. Can you elaborate on the apparent contradiction between your assumption that this is actually a food area (identified earlier in the paper) or is it one driven by high colour-sensitivity and therefore might be confounded with highly colourful food images?\n- Figure 7: I don’t understand how these values are calculated and the y-axis is not labelled. I wanted to ask you to be more precise on what “spatial similarity” means in terms of the method use to derive the results.\n- Not quite sure the logic for why the procedure described with the learning-free distillation module results in denoised features. Why does applying a transform, running an image through an encoder, then projecting the results back again, result in denoised features? It seems to work but I just felt like the method was explained in the \"how\" and not \"why\" it works. The paper could be made stronger by adding in some intuition in the relevant section.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "- The authors show alternatives to high-frequency artefact removal that are more computationally efficient than other solutions out there, namely by recognising the existing common method of adding register tokens in ViTs can be replaced with their proposed learning-free distillation module, which avoids the need for (computationally demanding) additional training.\n- The authors show that this method is a good way to target high confounds in naturalistic images by studying things like colour confounding with food images, luminance modulations in inside/outside scenes, depth information in varying images containing varying “reach spaces” etc.\n- The paper highlights potential equivalence between representations derived under varying training algorithms and questions what is being learned and the generalisation of features. I found this to be timely as recent discussions with LLMs and the utility of methods like BrainScore have been brought into question. I think we need to be asking these questions across the board and I appreciate this final section of the paper that raises this issue.\n- Visualisations are very nice and clear, making some more computationally dense text a bit more clear with reference to specific examples that underlie various findings (e.g. the effects of no language supervision in DINO and the downstream affects on the derived image maps)", "weaknesses": "- PDF is nearly 50 pages long. While ICLR allows for 10 main pages and unlimited supplementary information, the density of what is provided is a bit on the extreme side. In light of this, I still found myself searching for implementational information that I felt was missing and could be added in a revised version (see *Questions*)\n- I reformulated many of the comments I first wrote as weaknesses into more directed questions in the section below, hoping that these points might be more easily addressable and will hopefully make it into the camera-ready version such that readers have a better experience of the paper by benefiting from some of the points of clarity that I ran into while reading.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "P8iili8mb7", "reviewer": "Reviewer_dUkJ", "summary": "The authors proposed a method called BrainSAIL to deal with the problem of multiple categories co-existing in the natural images, combining the image embeddings and dense visual features and further identifying specific sub-regions of each image, which reflects the selectivity patterns in different areas of the higher visual cortex. BrainSAIL can realize the semantic attributes and positioning of related objects in complex images, which helps decompose the high-level visual representation of the human brain.", "questions": "DINO trained without language guidance shows high sensitivity to low-level visual features, does this hurt its performance on high-level semantic selection?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tBrainSAIL can obtain dense embeddings without artifacts and additional training through an efficient learning distillation module for high-throughput presentation of the visual cortex on large datasets.\n2.\tCompared to other studies that used simplified images, BrainSAIL can isolate specific image regions that activate different cortical areas when viewing natural scenes, providing a more complete description of how real-world visual stimuli are represented and processed.\n3.\tThe experiments are very sufficient, especially in the appendix.\n4.\tThis work demonstrated that an artifact-free dense feature map can be derived to explore high-throughput selectivity in the visual cortex.", "weaknesses": "1.\tThis method has a high model dependency (e.g., CLIP, DINO, SigLIP). While this improves efficiency, it limits flexibility to adapt feature extraction to the specific needs of neural regions, potentially constraining the granularity of semantic selectivity.\n2.\tThe method's fMRI training uses specific datasets (e.g., NSD), which may introduce bias, limiting generalizability across different populations or visual tasks—especially when the dataset’s images or semantic information don't cover the full range of possible visual stimuli.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "H48qSLBDlk", "reviewer": "Reviewer_PK4z", "summary": "The paper introduces a method for mapping the brain activity obtained using fMRI to semantically meaningful features in image space using vision transformer backbones and a parameter-free distillation process for denoising.", "questions": "How is it possible to verify that the method is not dominated by the foundation model representation of image rather than actually representing brain semantic representations?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The method was explained clearly\n- Literature review seems exhaustive\n- Claims SOTA (although I couldn't verify this)", "weaknesses": "-Abstract strats with features of BrainSAIL (first few sentences) before getting into WHAT BrainSAIL is.\n\n-It is not clear to me after training the backbone and distillation, how much of the output result is actually driven by brain data or completely rely on features of the image learned by the frozen foundation model.\n\n-SOTA was claimed but I wasn't able to find any comparison to other relevant methods in the paper.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)