Brain Mapping with Dense Features: Grounding Cortical Semantic Selectivity in Natural Images With Vision Transformers

## Abstract

Abstract Advances in large-scale artificial neural networks have facilitated novel insights into the functional topology of the brain. Here, we leverage this approach to study how semantic categories are organized in the human visual cortex.
To overcome the challenge presented by the co-occurrence of multiple categories in natural images, we introduce BrainSAIL ( S emantic A ttribution and I mage L ocalization), a method for isolating specific neurally-activating visual concepts in images. BrainSAIL exploits semantically consistent, dense spatial features from pre-trained vision models, building upon their demonstrated ability to robustly predict neural activity. This method derives clean, spatially dense embeddings without requiring any additional training, and employs a novel denoising process that leverages the semantic consistency of images under random augmentations. By unifying the space of whole-image embeddings and dense visual features and then applying voxel-wise encoding models to these features, we enable the identification of specific subregions of each image which drive selectivity patterns in different areas of the higher visual cortex. This provides a powerful tool for dissecting the neural mechanisms that underlie semantic visual processing for natural images. We validate BrainSAIL on cortical regions with known category selectivity, demonstrating its ability to accurately localize and disentangle selectivity to diverse visual concepts. Next, we demonstrate BrainSAIL’s ability to characterize high-level visual selectivity to scene properties and low-level visual features such as depth, luminance, and saturation, providing insights into the encoding of complex visual information. Finally, we use BrainSAIL to directly compare the feature selectivity of different brain encoding models across different regions of interest in visual cortex. Our innovative method paves the way for significant advances in mapping and decomposing high-level visual representations in the human brain.

## 1 Introduction

Understanding how the human brain processes and represents visual information from natural experience is a fundamental challenge in neuroscience. The vast majority of our knowledge of the visual system comes from tightly controlled experiments using simplified, hand-crafted images or, at best, real-world photographs of objects against noise backgrounds. Although this paradigm has revealed a pattern of preferential neural responses to semantic categories such as faces, places, bodies, words, objects, and food (Sergent et al., ; Allison et al., ; McCarthy et al., ; Kanwisher et al., ; Aguirre et al., ; Epstein & Kanwisher, ; Downing et al., ; Grill-Spector, ; Malach et al., ; Khosla et al., ; Pennock et al., ; Jain et al., ), the visual world we actually experience consists of rich, complex scenes containing many co-occurring objects, textures, and contextual associations (Simoncelli & Olshausen, ; Torralba & Oliva, ). As such, using minimal or single-object stimuli narrows the space of hypothesis testing and limits the ecological relevance of any conclusions, leaving us with an incomplete characterization of how the brain represents and processes real-world visual stimuli.

Recent developments in computer vision models trained on web-scale datasets have enabled learning rich multimodal representations that capture semantic concepts
in a human-aligned manner (Conwell et al., ; Wang et al., ). In this work, we introduce a novel methodology that leverages the power of such models to decompose selectivity patterns in visual cortex by analyzing responses to dense, localized semantic features present in naturalistic images: Semantic Attribution and Image Localization (“BrainSAIL”). BrainSAIL allows us to isolate the specific image regions that activate different cortical areas when viewing naturalistic scenes. This method allows us to focus on selectivity within complex naturalistic images, thereby enabling a richer decomposition grounded in the full semantic complexity of natural visual experiences.

The core of BrainSAIL involves extracting spatially dense semantic embeddings from images using state-of-the-art models such as CLIP, DINO, or SigLIP (Radford et al., ; Caron et al., ; Zhai et al., ). These embeddings bridge the traditionally disparate domains of raw vision data, dense deep semantic features, and measured neural responses. Within this rich embedding space, we can isolate and identify the specific visual features and corresponding image regions that drive selectivity effects in different cortical areas during perception of naturalistic visual scenes. By concurrently modeling localized semantic information, high-level semantic categories, and observed brain activity patterns, BrainSAIL can tease apart the image-level visual drivers of neural tuning preferences across higher visual areas. We validate this dense feature mapping method on a large-scale fMRI dataset consisting of human participants viewing many thousands of diverse natural images that span a wide range of semantic categories and visual statistics (Allen et al., ).

BrainSAIL’s dense embedding framework offers an interpretable view of feature representations across visual regions of the brain. Critically, this view explicitly grounds neural selectivity to localized semantic characteristics inherent in real-world visual experiences. First, we demonstrate the utility of our model for natural images applied to known category-selective regions of the cortex. Second, we show that our model can be used to identify the preference of brain regions sensitive to scene statistics. Finally, we use our model to compare and contrast the feature selectivity for different vision foundation models.
In sum, the dense semantic grounding realized in BrainSAIL enables exciting new directions towards understanding and modeling high-level visual representation in humans.

## 2 Related Work

A growing body of work leveraging computational modeling and machine learning has explored semantic representation in the higher visual cortex. Approaches include generative image models (Ratan Murty et al., ; Gu et al., ; Pierzchlewicz et al., ; Luo et al., ; ) and the decoding of visual stimuli (Takagi & Nishimoto, ; Chen et al., ; Doerig et al., ; Ferrante et al., ; Liu et al., ; Scotti et al., ). These diverse studies are united by their consideration of the stimulus image as a whole, primarily focusing on the global information contained within the image rather than the individual scene components. In contrast, the method we introduce explicitly decomposes an image into its semantic components, enabling the identification of individual, semantically meaningful activating concepts within complex natural images.

#### Semantic Representation in the Visual Cortex.

Using hand-crafted image stimuli, functional mapping studies have identified regions in the human brain that respond preferentially to stimuli representing distinct semantic concepts such as faces, places, bodies, words, objects, and food (Desimone et al., ; Sergent et al., ; Allison et al., ; McCarthy et al., ; Kanwisher et al., ; Gauthier & Tarr, ; Aguirre et al., ; Epstein & Kanwisher, ; Aguirre et al., ; O’Craven & Kanwisher, ; Nakamura et al., ; Aminoff et al., ; Downing et al., ; Cohen et al., ; Grill-Spector, ; Khosla et al., ; Pennock et al., ; Jain et al., ). One limitation of this simplified approach is that it may not fully capture the contextual complexity of natural vision (Gallant et al., ; Mahon, ). Addressing this concern, recent work on image-computable encoders has enabled computational tests of visual selectivity using naturalistic images  (Naselaris et al., ; Huth et al., ; Yamins et al., ; Eickenberg et al., ; Wen et al., ; Kubilius et al., ; Popham et al., ; Conwell et al., ; Wang et al., ; Luo et al., ; Prince et al., ; Luo et al., ; Yang et al., ; ; Efird et al., ). Building on this work, our method leverages state-of-the-art brain encoding backbones based on vision transformers (Dosovitskiy et al., ; Wang et al., ) to further explore finer-grained semantic representation in visual cortex.

#### Visual Contrastive Representation Learning.

Self- or weakly-supervised vision models that use contrastive  (Xing et al., ; Schultz & Joachims, ; Chopra et al., ; Sohn, ; Wu et al., ; Musgrave et al., ) and masked prediction objectives (Pathak et al., ; Kolesnikov et al., ; Chen et al., ; Zhao et al., ; Li et al., ; Zhou et al., ) are scalable and can be trained on massive, diverse datasets to achieve high zero-shot performance on downstream tasks. Contrastive models such as CLIP, DINO, and SigLIP demonstrate strong classification performance without further fine-tuning (Radford et al., ; Caron et al., ; Oquab et al., ; Zhai et al., ). Models that jointly train on language and vision (CLIP/SigLIP) can also classify images using text-based descriptions without fine-tuning. Interestingly, this high level of performance is mirrored in the fact that contrastive models show high performance for predicting neural responses in visual cortex when paired with linear probes (Conwell et al., ; Wang et al., ).

#### Exploring the Brain with Foundation Models.

There has been strong interest in leveraging generative models for decoding (reconstructing) visual stimuli conditioned on brain activations either directly or via intermediate language-based captions (Kamitani & Tong, ; Han et al., ; Seeliger et al., ; Shen et al., ; Ren et al., ; Takagi & Nishimoto, ; Chen et al., ; Lu et al., ; Ozcelik & VanRullen, ; Doerig et al., ; Ferrante et al., ; Liu et al., ; Mai & Zhang, ; Scotti et al., ). A related approach generates novel stimuli that are posited to best to activate a target brain region (as opposed to reconstructing the original stimulus) (Walker et al., ; Bashivan et al., ) with recent attempts utilizing GANs or Diffusion models to constrain the synthesized output (Ponce et al., ; Ratan Murty et al., ; Gu et al., ; Luo et al., ; ). While these models have shown positive results, they all rely on images as a whole, whereas BrainSAIL seeks to disentangle complex images into their semantically meaningful components and localize those parts of the image that elicit activation for different brain voxels or regions.

## 3 Methods

Figure: Figure 1: The BrainSAIL framework leverages dense visual features. (a) An fMRI encoder learns a map from images to voxel-wise activations in the brain. Encoders leveraging frozen foundation models based on vision transformers (ViTs) with voxel-wise adapters are currently the highest accuracy models for brain prediction (Conwell et al., ; Wang et al., ). (b) Given an image and a ViT backbone for the fMRI encoder, we modify the backbone to output dense features. The dense backbone is wrapped inside of a Learning-Free Distillation Module. This module takes an image $\mathcal{I}$ and $2D$ image coordinates $\mathcal{C}$, and generates transformed images and coordinates $(\mathcal{I}_{i},\mathcal{C}_{i})$ for a given transform $\theta_{i}$. The dense features and transformed coordinates are provided to a denoising module to generate clean dense features. A voxel-wise adapter then generates dense relevance maps which highlight the image regions activating the voxel. (c) Using CLIP ViT-B/16 with the latest NACLIP adapter, we show relevance maps using the CLIP text encoder. The original, raw features are highly noisy and contain artifacts, while the distilled features are localized to the relevant semantic components with high accuracy. Note that we achieve state-of-the-art open vocabulary CLIP-based segmentation results using our method.
Refer to caption: x1.png

Our aim is to generate spatial attribution maps for arbitrary voxels in the higher visual cortex. Unlike the early visual cortex, which is believed to be primarily selective for “simple features” (Stork & Wilson, ), the higher visual cortex exhibits semantic selectivity – a pattern that, at present, is best predicted by deep networks (Conwell et al., ; Wang et al., ). As illustrated in Figure 1, to create spatial attributions maps for brain voxels, we first train voxel-wise fMRI encoders to map images to brain activations. Second, we derive dense features from pre-trained vision transformers (ViT) used as the backbone for these encoders. Third, we demonstrate that an artifact-free dense feature map can be derived for high-throughput exploration of selectivity with the visual cortex.

### 3.1 Image-to-Brain Encoders for the Higher Visual Cortex

A voxel-wise image-computable fMRI encoder is a model $F_{\phi}$ that predicts fMRI activations (betas) for $B\in\mathbb{R}^{1\times N}$ where $N$ represents the number of voxels in the brain. The encoder is conditioned on image input $\mathcal{I}\in\mathbb{R}^{H\times W\times 3}$, where $F_{\phi}(\mathcal{I})\Rightarrow B$. Recent work has demonstrated that encoders that rely on features extracted from large vision foundation models achieve excellent predictive performance, where higher visual cortex is best predicted by deeper layers in the model (Wang et al., ). In this setting, the backbone model is usually frozen, while a per-voxel adapter typically parameterized as a linear layer is trained to map from network features to voxel activations. In that we focus on the higher visual cortex exclusively, we utilize a two component design for our encoder: (1) a frozen vision foundation model backbone $\text{G}(\mathcal{I})$ which outputs a $R^{1\times M}$ dimension embedding vector for each image; (2) a per-voxel adapter parameterized as a linear probe with weight $W\in\mathcal{R}^{M\times N}$ and bias $b\in\mathcal{R}^{1\times N}$, which takes as input a unit-norm image embedding.

$$ $\displaystyle\left[\frac{\text{G}_{\text{img}}(\mathcal{I})}{\lVert\text{G}_{ \text{img}}(\mathcal{I})\rVert_{2}}\times W+b\right]\Rightarrow B$ (1) $$

It should be noted that BrainSAIL is not restricted to linear probes, and can work with arbitrary voxel-wise parameterizations, including MLPs. Linear probes are used here as they are widely adopted in fMRI encoder literature and empirically achieve good performance. BrainSAIL is compatible with any Vision Transformer (ViT)-based model, making it readily applicable to the vast majority of modern visual foundation models which predominantly employ ViT architectures. Additional results are presented in the supplemental. We train our model with MSE loss, and evaluate the encoder on the test set. In Figure 6 we show that our encoder achieves state-of-the-art $R^{2}$.

### 3.2 Deriving Dense Features from ViT backbones

The emergence of vision models trained on a contrastive image-text objective has fueled interest in zero-shot open-vocabulary image classification methods. For example, CLIP has shown that images can be classified without foreknowledge of the test time classes during training; instead the category of interest can be described using language during test time. Of late, this capability has been extended from classification to segmentation. Compared to methods that require human annotation (Li et al., ) and perform poorly on out-of-distribution images (Jatavallabhula et al., ; Kerr et al., ), these new methods require no further training and directly extract dense features that lie in the same space as the image/text embedding.

Figure: Figure 2: The Learning-Free Distillation Module. (a) Given an image, we generate image-space coordinates $(u,v)$ for each pixel. We then randomly sample from $\theta_{1...n}$, where $\theta_{i}$ has vertical/horizontal offset, and left-right flips. The augmented images are provided to a frozen backbone with dense adapter. The features are projected to the original image space via an inverse transform $\mathcal{T}^{-1}(\theta_{i})$. (b) UMAP visualization of the dense features. The same fitted basis is used for both visualizations. (c) With CLIP, we can perform zero-shot text queries. Note the artifacts above the bird’s head. In practice artifact location is different for each image. The distilled results are significantly better.
Refer to caption: x2.png

These dense feature extraction methods operate by modifying the last self-attention (SA) block within the typical ViT architecture (MaskCLIP, Zhou et al. (); SCLIP, Wang et al. (); NACLIP, Hajimiri et al. ()). For vision models of this sort trained on a contrastive objective, the output is composed of a single [CLS] token, which is supervised using a contrastive loss; and numerous patch tokens which correspond to specific spatial locations. Let $(q_{i},k_{i},v_{i})$ be the query, key, value features respectively for a single image patch $i$, with a total of $m$ spatial patches. For a given patch $j$ at the final layer, where f denotes any function applied to the [CLS] after the last self-attention, the [CLS] token and each dense token is a convex combination of $v$ features:
$\displaystyle\text{Out_Orig}_{j}=f\left({\sum_{k=1}^{m}\left[\text{softmax}(
\frac{q_{j}k^{T}}{C})_{j}\cdot v_{k}\right]}\right)\quad\text{Out_Mask}_{j}=f
(v_{j})\quad\text{Out_NA}_{j}=f\left({\sum_{k=1}^{m}\left[\text{softmax}(
\frac{q_{j}q^{T}+\omega_{j}}{C})_{j}\cdot v_{k}\right]}\right)$
(2)
MaskCLIP proposes to directly remove the convex re-weighting and output the value feature for each patch token directly. SCLIP and NACLIP reintroduce the weighting to reduce output artifacts, but modify it with correlative self-attention (CSA); or by using CSA with a spatial attentive bias $\omega$. Here, we utilize NACLIP as the dense adaptor for CLIP. The other two backbones in Section 4.4 use an updated ViT architecture with “register tokens” (Darcet et al., ). As these have not been explored in the context of CSA, we utilize MaskCLIP as the dense adaptor.

Figure: Algo 1: Learning-Free Feature Distillation

### 3.3 Learning-Free Feature Distillation

As only the [CLS] is supervised in these contrastive models, as shown in Figures 1 and 2, the extracted dense embeddings often have artifacts – even when using the latest NACLIP method which seeks to reduce artifacts. While methods such as Darcet et al. () improve spatial consistency via architectural improvements, they require training the model with architecture modifications that are computationally costly. Consequently, in order to facilitate high-throughput characterization of the visual cortex over large datasets, we propose an efficient learning-free distillation module. Given an image $\mathcal{I}$, we first generate $n$ augmentation parameters $\theta_{1...n}$, where $\theta_{i}$ consists of a horizontal/vertical offset $(u_{i},v_{i})$ and horizontal $\text{flip}_{i}\in\{0,1\}$. We further generate the image space coordinates $\mathcal{C}=(\text{u-coord},\text{v-coord})$, where $u\in[0,1]$ goes from top-to-bottom, while $v\in[0,1]$ goes left-to-right. We describe our full transform in Algorithm 1. Our method distills a clean semantic map, as visual semantics are equivariant to shift and horizontal flips. We note that averaging over the number of augmentation is extracting an optimal embedding under mean squared error (squared euclidean). Let $\vec{p^{*}}$ be the optimal embedding under MSE for a given patch, and $\vec{p}_{i}$ with $i\in\{1...n\}$ be the feature candidates under image augmentation:

$$ $\displaystyle\vec{p^{*}}$ $\displaystyle=\min_{\hat{p}}\left(\sum_{i=1}^{n}\left\lVert\vec{p}_{i}-\hat{p} \right\rVert_{2}^{2}\right)=\min_{\hat{p}}\left(\left\lVert\vec{p}_{1}-\hat{p} \right\rVert_{2}^{2}+...+\left\lVert\vec{p}_{n}-\hat{p}\right\rVert_{2}^{2}\right)$ (3) $\displaystyle=\min_{\hat{p}}\left(\vec{p_{1}}^{T}\vec{p_{1}}-2\vec{p_{1}}^{T} \hat{p}+\hat{p}^{T}\hat{p}+...+\vec{p_{n}}^{T}\vec{p_{n}}-2\vec{p_{n}}^{T}\hat {p}+\hat{p}^{T}\hat{p}\right)\quad\text{omitting }\vec{p_{i}}^{T}\vec{p_{i}}$ (4) $\displaystyle=\min_{\hat{p}}\left(n\cdot\hat{p}^{T}\hat{p}-2\sum_{i=1}^{n}( \vec{p_{i}}^{T}\hat{p})\right)=\min_{\hat{p}}\left(n\cdot\hat{p}^{T}\hat{p}-2n \sum_{i=1}^{n}((1/n)\cdot\vec{p_{i}}^{T}\hat{p})\right)$ (5) $$

The objective can be expressed as $\left\lVert\hat{p}-(1/n)\sum\vec{p_{i}}\right\rVert_{2}^{2}\geq 0$, then $\vec{p^{*}}=(1/n)\sum\vec{p_{i}}$.

In Table 1, we compare pre- and post- smoothing results. Under the Pearson metric, which does not assume prior category knowledge, smoothing yields the best performance in three of four datasets. We apply the voxel-wise adapters to the per patch dense features to derive the final relevance map.

**Table 1: Open-vocabulary segmentation with dense CLIP features. We validate the effectiveness of our learning-free smoothing approach on segmentation datasets in a zero-shot setting (without any training). This performance is state-of-the-art for open vocabulary segmentation. Note mIoU scores are multiplied by 100. Our smoothing improves the results.**
|  | ADE20k | COCO Object | COCO Stuff | VOC20 |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | mIoU$\uparrow$ | Pearson$\uparrow$ | mIoU$\uparrow$ | Pearson$\uparrow$ | mIoU$\uparrow$ | Pearson$\uparrow$ | mIoU$\uparrow$ | Pearson$\uparrow$ |
| \addstackgapSCLIP | 16.45 | 0.308 | 33.52 | 0.353 | 21.95 | 0.309 | 81.54 | 0.551 |
| NACLIP | 17.69 | 0.425 | 33.14 | 0.418 | 22.58 | 0.393 | 77.09 | 0.473 |
| +Distilled | 18.19 | 0.443 | 34.15 | 0.435 | 23.08 | 0.405 | 79.09 | 0.489 |

## 4 Results

Figure: Figure 3: Joint dimensional reduction of higher visual cortex encoder weights and images using BrainSAIL. We use a UMAP to perform visualization of the encoder weights. This same UMAP basis is reused for images. (a) Cortical flatmap of S1. Note that the overlaid white region outlines and labels were derived from functional localizer data collected independently from the visualized UMAP results. (b) Embeddings from novel images are computed with BrainSAIL and transformed using the fMRI UMAP. For each quartet of images, the content is as follows – Top left: Original RGB image; Top right: Dimension reduction of BrainSAIL embeddings for the image; Bottom: Two text queries using CLIP text branch showing language-indicated relevance results. (c) UMAP results on an inflated view of the brain for S1. (d) UMAP results on cortical flatmaps for S2, S5 and S7. These results demonstrate that BrainSAIL can effectively localize semantically meaningful components of natural images and map them to appropriate brain regions. The cortical maps show color-coded mappings that align well with functionally-defined regions: body regions (EBA), face regions (FFA/aTL-faces), place regions (RSC/OPA/PPA), and food regions (yellow). Note that the food regions have been identified as flanking FFA by Jain et al. (), but we do not have independent functional localizer data for food for these subjects.
Refer to caption: x3.png

We utilize BrainSAIL to localize the semantic selectivity of different brain regions and demonstrate that the relevance maps are interpretable throughout the brain and correlate well with the known category-selective regions. We then explore the selectivity of higher visual cortex with respect to localized scene structure and image properties. Finally, we compare and contrast the localization results from three different vision foundation models. These results establish BrainSAIL as a novel technique for mapping and understanding the semantics of visual representations in the brain.

**Table 2: CLIP text alignment for each category selective brain region. For each category selective brain region, we take the top-100 images from the NSD test set that elicit the highest fMRI response for each region. We then use BrainSAIL to compute the relevance maps for the top-100 images for each region. For each image, additional relevance maps are computed using the CLIP text encoder with text prompts from the five relevant categories. The text prompt with the highest Pearson correlation to the BrainSAIL relevance map is recorded as the category for that image. Units in %.**
| Region | Faces | Places | Bodies | Words | Food |  |  |  |  |  |  |  |  |  |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
|  | S1 | S2 | S5 | S7 | S1 | S2 | S5 | S7 | S1 | S2 | S5 | S7 | S1 | S2 | S5 | S7 | S1 | S2 | S5 | S7 |
| \addstackgapFace | 46 | 48 | 54 | 40 | 1 | 1 | 2 | 2 | 12 | 11 | 12 | 13 | 9 | 11 | 12 | 11 | 1 | 2 | 1 | 5 |
| Places | 1 | 1 | 1 | 3 | 76 | 80 | 89 | 75 | 0 | 2 | 3 | 2 | 9 | 12 | 11 | 9 | 8 | 7 | 10 | 7 |
| Bodies | 26 | 16 | 21 | 27 | 5 | 1 | 0 | 1 | 50 | 41 | 55 | 48 | 15 | 13 | 21 | 30 | 9 | 5 | 1 | 5 |
| Words | 1 | 7 | 3 | 8 | 6 | 3 | 2 | 9 | 8 | 6 | 7 | 7 | 38 | 28 | 26 | 23 | 16 | 13 | 17 | 35 |
| Food | 26 | 28 | 21 | 22 | 12 | 15 | 7 | 13 | 30 | 40 | 23 | 30 | 29 | 36 | 30 | 27 | 66 | 73 | 71 | 48 |

### 4.1 Setup

We use the Natural Scenes Dataset (NSD;  Allen et al. ()), the largest 7T fMRI dataset of human visual responses, focusing on four subjects (S1, S2, S5, S7) who viewed the full 10,000 image set (a subset of COCO images) three times each. fMRI activations (betas) were derived using GLMSingle (Prince et al., ) and normalized per session ($\mu=0,\sigma^{2}=1$). Responses to repeated images were averaged. A brain encoder for each subject was trained on $\sim 9000$ unique images per subject, with the remaining $\sim 1000$ images viewed by all subjects being used for $R^{2}$ validation as the test set. Supplementary results for other subjects are included in the appendix.
Face, place, body, and word regions were defined using independent category localizer data from NSD with a threshold of $t>2$ (Stigliani et al., ). Food regions were defined using masks provided by Jain et al. ().

We train three encoders based on different neural network backbones. For all three, we utilize the ViT-Base model size. (1) For CLIP, we utilize OpenAI’s official ViT-B/16 weights. This is a network trained on an infoNCE contrastive image-text objective. (2) For DINO, we utilize the latest official DINOv2 ViT-B/14+reg, and is a network trained on image-only self-supervision (Darcet et al., ). (3) For SigLIP, we utilize NVIDIA’s implementation based on RADIOv2.5 ViT-B/16 (Ranzinger et al., ), as the original Google variant used a non-standard architecture. SigLIP utilizes a pairwise non-contrastive image-text objective (Zhai et al., ). All fMRI encoders are trained using MSE loss, with the backbone frozen. We validate the test time $R^{2}$ in Figure 6 and find that we achieve state-of-the-art results similar to Wang et al. () and Luo et al. (). We use CLIP for Sections 4.2 and 4.3, as it is the most widely used backbone in fMRI literature. We use $51$ augmentation steps unless otherwise noted.

Figure: Figure 4: Grounding results using BrainSAIL. We visualize the top test set images as predicted by the CLIP fMRI encoder for each category selective region. For each image, we also visualize the image-wise UMAP for the distilled dense features. Note the UMAP basis here is computed imagewise, and not shared with Figure 3. For each image, we further visualize the feature relevancy map for the category selective voxels illustrating that this method extracts the semantically relevant regions in complex compositional images.
Refer to caption: extracted/5908168/fig_pdf/brainsail.png

### 4.2 Image Factorization using the Brain

To explore how different areas in higher visual cortex align to different image parts we apply UMAP (McInnes et al., ) with an angular metric to linear brain weights and apply the same UMAP basis to dense features as produced by BrainSAIL. Note that during dimensionality reduction we do not utilize any cortex category masks from NSD – the region of interest outlines on the cortex in Figure 3a are for visualization purposes only and are derived from independent NSD functional localizers. As shown in Figure 3, we find that the factorization of the brain is well aligned to pre-identified functional regions, and broadly segments the cortex into axes along “people”, “scenes” and “food”. In particular, place regions, including the retrosplenial cortex (RSC), occipital place area (OPA), and parahippocampal place area (PPA), show selectivity for scene components (magenta). People regions, including the extrastriate body area (EBA), fusiform face area (FFA), occipital face area (OFA), show selectivity for face and body parts in the image (Green-Blue). Finally, we find that the recently identified food region that roughly surrounds FFA (Yellow) Khosla et al. (); Pennock et al. (); Jain et al. () strongly corresponds to food in images. These results establish that BrainSAIL can be used to characterize higher-level selectivity to individual semantic categories in complex natural images without prior knowledge of their semantic selectivity.

We further quantify the feature relevance maps for broad category selective regions in Figure 4 and Table 2. We use the brain encoder to predict the top-5 images for the place/word/face/body regions, and the top-10 images for the food region. We find that our method can effectively localize the objects relevant to each category- selective brain region. Note that the word region is known to have cross-selectivity to faces (Mei et al., ) and food (Khosla & Wehbe, ).

### 4.3 Cortex Selectivity to Image Features

Going beyond semantic categories, we seek to explore the low- and mid-level image feature correlates that correspond to different brain regions. Prior work explored this by training a convolutional encoder on each NSD subject, which is limited to $\sim 10,000$ images each (Sarch et al., ). One concern is that using a small dataset with a convolutional backbone can lead to overfitting to the dataset’s specific features and exacerbate the inherent biases of convolutional networks. To address this limitation, our method leverages vision transformers trained on massive datasets of hundreds of millions of images, thereby avoiding the hard-coded inductive biases present in CNNs (Raghu et al., ). We visualize BrainSAIL feature dissection results in Figure 5. Our method can successfully identify the known scene selective regions (RSC/OPA/PPA) as preferring high depth, and is successful even in OPA where Sarch et al. () fails. We believe this is likely because OPA processes higher-level associative content and affordances (Bonner & Epstein, ; Aminoff & Tarr, ).
Similarly, we identify the region surrounding FFA as being selective to high color saturation, which correspond to the food regions identified by Jain et al. () and others. In OPA, we identify a split in color luminance preference, which is similar to the indoor/outdoor preferring regions identified by Lescroart & Gallant (), Peer et al. (), and  Luo et al. (). These results demonstrate that our method can identify fine-grained selectivity with more broadly characterized brain regions.

Figure: Figure 5: Feature correlates with BrainSAIL. We visualize the depth, color saturation, and color luminance (brightness) correlates for each brain region using BrainSAIL . (a) The scene selective regions, retrosplenial cortex (RSC), parahippocampal place area (PPA), and occipital place area (OPA) are all identified as having a preference for high depth. (b) On the ventral surface, we identify two stripes on each hemisphere, surrounding FFA with high saturation preference. These are the same brain regions identified by Jain et al. () as being food selective. (c) In OPA, we identify an anterior/posterior split, where one region has high color luminance preference, and the other has low color luminance preference. This are the same regions identified by Luo et al. () as being outdoor/indoor selective.
Refer to caption: x4.png

### 4.4 Are Brain Encoders Equivalent?

Recent high-performing models such as CLIP, DINO, and SigLIP differ in their training objectives, architectures, and datasets: CLIP employs a contrastive image-language objective, DINO utilizes a self-supervised image loss without explicit linguistic guidance, and SigLIP leverages a non-contrastive pairwise image-language loss. Despite these differences, when employed as the backbone for fMRI encoders, these models exhibit similar performance in predicting brain responses, achieving comparable $R^{2}$ values on the test set as shown in Figure 6. This observation raises an important question about the nature of each model’s learned features and their alignment with one another: Do these models converge upon similar feature representations (Chen & Bonner, ) for category selective brain regions despite their varied training paradigms?

To investigate the representational differences between the models, we perform BrainSAIL analysis for scene, face, and food-selective brain regions and qualitatively visualize the results in Figure 8. While all three models exhibit broad similarities in their grounding maps, DINO, trained without language supervision, demonstrates a stronger sensitivity to low-level visual features compared to CLIP and SigLIP (Wang et al., ). This is evident in the food region (Figure 8), where DINO’s grounding map for a pizza image excludes the toppings and misses differently colored vegetables on a metal plate, suggesting a focus on color and texture rather than the concept of “food” itself. Similarly, in the face region, DINO’s grounding map exhibits less reliance on semantically relevant features such as eyes, nose, and mouth. We hypothesize that this greater sensitivity to visual features in DINO stems from its lack of language guidance during training, preventing it from learning the higher-level semantic correlations that link visually disparate parts and objects within a category. As high-performing “proxy models” of visual brain representation (Leeds et al., ), these and other underlying model characteristics – architecture, training objective, training dataset, etc. – are important considerations for developing more robust encoding models that can bridge the gap between artificial and biological vision systems.

Figure: Figure 6: Comparing the brain prediction performance for different encoder backbones. (a) We validate each encoder $R^{2}$ on a test set and find that all three models achieve very high performance (comparable to Wang et al. ()). (b) The voxel-wise correlation of test set $R^{2}$ for the three models. CLIP and SigLIP, which rely on language supervision, achieve higher performance than DINO (which trained via self-supervision with images).
Refer to caption: x5.png

Figure: Figure 7: Model similarity across ROIs. Brain encoder backbone spatial similarity for the ground truth top-$100$ images from the test set for each category-selective brain region. A $\star$ denotes a domain-defined network of regions encompassing multiple ROIs. CLIP and SigLIP relevance maps are more similar to one another than either is to DINO. Error bars indicate standard error across the $100$ images.
Refer to caption: x6.png

Figure: Figure 8: Comparing different brain encoder backbones with BrainSAIL. Visualization of the top test set images for the place, face, and food category-selective brain regions as predicted by CLIP, SigLIP, and DINO. While all models show broadly similar feature relevance for a given brain area, there are important differences. DINO, with no language supervision, exhibits greater sensitivity to visual similarity, at the cost of semantic coherence.
Refer to caption: extracted/5908168/fig_pdf/saliency.png

## 5 Discussion

#### Limitations and Future Work.

BrainSAIL achieves strong localization performance and benefits from a pre-trained vision transformer, reducing reliance on the fMRI dataset for backbone training. However, it is still necessary to train the fMRI encoder on these data, and thus potential dataset biases in the human neural data and how it was collected can influence the learned representations and conclusions. Future work should explore training on larger and more diverse neural datasets to mitigate this limitation and enhance the generalizability of our findings.

#### Conclusion.

We propose BrainSAIL, a method that leverages vision foundation models to interrogate which semantic components of complex natural images lead to the neural activation of specific regions of the brain. Based on the vision transformer architecture, we: (1) semantically attribute and localize relevant objects in complex compositional images; (2) jointly factorize images and semantically selective regions in the human brain; (3) identify the feature correlates of depth, saturation, and luminance that underlie semantic selectivity; (4) explicate differences in fMRI encoders that achieve similar overall brain prediction performance.
In toto, these results establish that BrainSAIL is a powerful new approach to data-driven explorations of the human higher visual cortex.

## Appendix A Appendix

Sections

- 1.
Additional encoder test set $R^{2}$ for CLIP, DINO, and SigLIP (A.1)
- 2.
Visualization of ground truth functional localizer statistics (A.2)
- 3.
Visualization of encoder weight UMAP for all subjects using CLIP backbone  (A.3)
- 4.
Visualization of encoder weight UMAP for all subjects using DINO backbone  (A.4)
- 5.
Visualization of encoder weight UMAP for all subjects using SigLIP backbone  (A.5)
- 6.
Additional visualization of fMRI grounding using CLIP backbone encoder  (A.6)
- 7.
Additional visualization of fMRI grounding using DINO backbone encoder  (A.7)
- 8.
Additional visualization of fMRI grounding using SigLIP backbone encoder  (A.8)
- 9.
Implementation details (A.9)

### A.1 Additional encoder test set R 2 superscript 𝑅 2 R^{2} italic_R start_POSTSUPERSCRIPT 2 end_POSTSUPERSCRIPT for CLIP, DINO, and SigLIP

Subjects 1, 2, 5, 7 are those that completed the full scan of $10,000$ images – where each image was viewed 3 times. They are also the subjects reported to have the highest noise ceiling in NSD (higher is better).

Figure: Figure S.1: Test set $R^{2}$ using different encoder backbones on subjects S1 & S2. (a) We validate each encoder $R^{2}$ on a test set and find that all three models achieve very high performance (comparable to Wang et al. ()). (b) The voxel-wise correlation of test set $R^{2}$ for the three models. CLIP and SigLIP, which rely on language supervision, achieve higher performance than DINO (which trained via self-supervision with images).
Refer to caption: x7.png

Figure: Figure S.2: Test set $R^{2}$ using different encoder backbones on subjects S3 & S4. (a) We validate each encoder $R^{2}$ on a test set and find that all three models achieve very high performance (comparable to Wang et al. ()). (b) The voxel-wise correlation of test set $R^{2}$ for the three models. CLIP and SigLIP, which rely on language supervision, achieve higher performance than DINO (which trained via self-supervision with images).
Refer to caption: x8.png

Figure: Figure S.3: Test set $R^{2}$ using different encoder backbones on subjects S5 & S6. (a) We validate each encoder $R^{2}$ on a test set and find that all three models achieve very high performance (comparable to Wang et al. ()). (b) The voxel-wise correlation of test set $R^{2}$ for the three models. CLIP and SigLIP, which rely on language supervision, achieve higher performance than DINO (which trained via self-supervision with images).
Refer to caption: x9.png

Figure: Figure S.4: Test set $R^{2}$ using different encoder backbones on subjects S7 & S8. (a) We validate each encoder $R^{2}$ on a test set and find that all three models achieve very high performance (comparable to Wang et al. ()). (b) The voxel-wise correlation of test set $R^{2}$ for the three models. CLIP and SigLIP, which rely on language supervision, achieve higher performance than DINO (which trained via self-supervision with images).
Refer to caption: x10.png

### A.2 Visualization of ground truth functional localizer statistics

In this section, we show the ground truth functional localizer t-statistic provided by NSD. As NSD does not provide a food localizer, we obtain the food mask from Jain et al. (). During the UMAP, we do not provide our method with the ground truth functional localizer information, and these t-statistics are provided here for reference only.

Figure: Figure S.5: Sematic category functional localizer statistics for S1. We visualize the ground truth functional localizer t-statistic provided by NSD. Higher indicates stronger selectivity to a semantic category.
Refer to caption: x11.png

Figure: Figure S.6: Sematic category functional localizer statistics for S2. We visualize the ground truth functional localizer t-statistic provided by NSD. Higher indicates stronger selectivity to a semantic category.
Refer to caption: x12.png

Figure: Figure S.7: Sematic category functional localizer statistics for S5. We visualize the ground truth functional localizer t-statistic provided by NSD. Higher indicates stronger selectivity to a semantic category.
Refer to caption: x13.png

Figure: Figure S.8: Sematic category functional localizer statistics for S7. We visualize the ground truth functional localizer t-statistic provided by NSD. Higher indicates stronger selectivity to a semantic category.
Refer to caption: x14.png

### A.3 Visualization of encoder weight UMAP for all subjects using CLIP backbone

We utilize OpenAI’s official ViT-B/16 as the encoder backbone, and visualize the UMAP as applied to the fMRI encoder weights and dense features.

Figure: Figure S.9: Encoder with CLIP backbone UMAP transform results for subjects S1-S4. We normalize all encoder weights to unit norm prior to UMAP, and use an angular metric for the fitting and projection stage. The UMAP is fitted on S1 and reused across all subjects and images. Both patch and voxel vectors are projected onto the space of natural images prior to transform using softmax weighted sum similar to BrainSCUBA. For each quartet of images, the content is as follows – Top left: Original RGB image; Top right: Dimension reduction of BrainSAIL embeddings for the image using CLIP features; Bottom: Two text queries using CLIP text branch showing language-indicated relevance results.
Refer to caption: x15.png

Figure: Figure S.10: Encoder with CLIP backbone UMAP transform results for subjects S5-S8. We normalize all encoder weights to unit norm prior to UMAP, and use an angular metric for the fitting and projection stage. The UMAP is fitted on S1 and reused across all subjects and images. Both patch and voxel vectors are projected onto the space of natural images prior to transform using softmax weighted sum similar to BrainSCUBA. For each quartet of images, the content is as follows – Top left: Original RGB image; Top right: Dimension reduction of BrainSAIL embeddings for the image using CLIP features; Bottom: Two text queries using CLIP text branch showing language-indicated relevance results.
Refer to caption: x16.png

### A.4 Visualization of encoder weight UMAP for all subjects using DINO backbone

We utilize Meta’s official DINOv2 ViT-B/14+reg as the encoder backbone, and visualize the UMAP as applied to the fMRI encoder weights and dense features. The UMAP method is nonparametric , and the output can depend on the random seed. Similar colors for UMAP applied to DINO encoder weights does not indicate any specific similarity to CLIP results in the prior section.

As DINO does not have a text encoder, we cannot visualize text-based localization probes.

Figure: Figure S.11: Encoder with DINO backbone UMAP transform results for subjects S1-S4. We normalize all encoder weights to unit norm prior to UMAP, and use an angular metric for the fitting and projection stage. The UMAP is fitted on S1 and reused across all subjects and images. Both patch and voxel vectors are projected onto the space of natural images prior to transform using softmax weighted sum similar to BrainSCUBA. For each pair of images, the content is as follows – Left: Original RGB image; Right: Dimension reduction of BrainSAIL embeddings for the image using DINO features.
Refer to caption: x17.png

Figure: Figure S.12: Encoder with DINO backbone UMAP transform results for subjects S5-S8. We normalize all encoder weights to unit norm prior to UMAP, and use an angular metric for the fitting and projection stage. The UMAP is fitted on S1 and reused across all subjects and images. Both patch and voxel vectors are projected onto the space of natural images prior to transform using softmax weighted sum similar to BrainSCUBA. For each pair of images, the content is as follows – Left: Original RGB image; Right: Dimension reduction of BrainSAIL embeddings for the image using DINO features.
Refer to caption: x18.png

### A.5 Visualization of encoder weight UMAP for all subjects using SigLIP backbone

We utilize Nvidia’s RADIOv2.5 ViT-B/16 implementation for SigLIP. Nvidia’s implementation was choosen as the original Google implementation did not utilize a ViT with a attention [CLS] token. We visualize the UMAP as applied to the fMRI encoder weights and dense features. The UMAP method is nonparametric , and the output can depend on the random seed. Similar colors for UMAP applied to SigLIP encoder weights does not indicate any specific similarity to DINO or CLIP results in the prior section.

Figure: Figure S.13: Encoder with SigLIP backbone UMAP transform results for subjects S1-S4. We normalize all encoder weights to unit norm prior to UMAP, and use an angular metric for the fitting and projection stage. The UMAP is fitted on S1 and reused across all subjects and images. Both patch and voxel vectors are projected onto the space of natural images prior to transform using softmax weighted sum similar to BrainSCUBA. For each quartet of images, the content is as follows – Top left: Original RGB image; Top right: Dimension reduction of BrainSAIL embeddings for the image using SigLIP features; Bottom: Two text queries using SigLIP text branch showing language-indicated relevance results.
Refer to caption: x19.png

Figure: Figure S.14: Encoder with SigLIP backbone UMAP transform results for subjects S5-S8. We normalize all encoder weights to unit norm prior to UMAP, and use an angular metric for the fitting and projection stage. The UMAP is fitted on S1 and reused across all subjects and images. Both patch and voxel vectors are projected onto the space of natural images prior to transform using softmax weighted sum similar to BrainSCUBA. For each quartet of images, the content is as follows – Top left: Original RGB image; Top right: Dimension reduction of BrainSAIL embeddings for the image using SigLIP features; Bottom: Two text queries using SigLIP text branch showing language-indicated relevance results.
Refer to caption: x20.png

### A.6 Additional Visualization of Grounding using CLIP backbone

Figure: Figure S.15: Grounding results using BrainSAIL for S1 CLIP backbone.. We visualize the top test set images according to NSD fMRI for each category selective region. For each image, we also visualize the image-wise PCA for the distilled dense features. Note the PCA basis here is computed imagewise. For each image, we further visualize the feature relevancy map for the category selective voxels illustrating that this method extracts the semantically relevant regions in complex compositional images.
Refer to caption: extracted/5908168/supp_stuff/fig_pdf/S1_grounding.png

Figure: Figure S.16: Grounding results using BrainSAIL for S2 CLIP backbone.. We visualize the top test set images according to NSD fMRI for each category selective region. For each image, we also visualize the image-wise PCA for the distilled dense features. Note the PCA basis here is computed imagewise. For each image, we further visualize the feature relevancy map for the category selective voxels illustrating that this method extracts the semantically relevant regions in complex compositional images.
Refer to caption: extracted/5908168/supp_stuff/fig_pdf/S2_grounding.png

Figure: Figure S.17: Grounding results using BrainSAIL for S5 CLIP backbone.. We visualize the top test set images according to NSD fMRI for each category selective region. For each image, we also visualize the image-wise PCA for the distilled dense features. Note the PCA basis here is computed imagewise. For each image, we further visualize the feature relevancy map for the category selective voxels illustrating that this method extracts the semantically relevant regions in complex compositional images.
Refer to caption: extracted/5908168/supp_stuff/fig_pdf/S5_grounding.png

Figure: Figure S.18: Grounding results using BrainSAIL for S7 CLIP backbone.. We visualize the top test set images according to NSD fMRI for each category selective region. For each image, we also visualize the image-wise PCA for the distilled dense features. Note the PCA basis here is computed imagewise. For each image, we further visualize the feature relevancy map for the category selective voxels illustrating that this method extracts the semantically relevant regions in complex compositional images.
Refer to caption: extracted/5908168/supp_stuff/fig_pdf/S7_grounding.png

### A.7 Additional Visualization of Grounding using DINO backbone

Figure: Figure S.19: Grounding results using BrainSAIL for S1 DINO backbone.. We visualize the top test set images according to NSD fMRI for each category selective region. For each image, we also visualize the image-wise PCA for the distilled dense features. Note the PCA basis here is computed imagewise. For each image, we further visualize the feature relevancy map for the category selective voxels illustrating that this method extracts the semantically relevant regions in complex compositional images.
Refer to caption: extracted/5908168/supp_stuff/fig_pdf/S1_grounding_DINO.png

Figure: Figure S.20: Grounding results using BrainSAIL for S2 DINO backbone.. We visualize the top test set images according to NSD fMRI for each category selective region. For each image, we also visualize the image-wise PCA for the distilled dense features. Note the PCA basis here is computed imagewise. For each image, we further visualize the feature relevancy map for the category selective voxels illustrating that this method extracts the semantically relevant regions in complex compositional images.
Refer to caption: extracted/5908168/supp_stuff/fig_pdf/S2_grounding_DINO.png

Figure: Figure S.21: Grounding results using BrainSAIL for S5 DINO backbone.. We visualize the top test set images according to NSD fMRI for each category selective region. For each image, we also visualize the image-wise PCA for the distilled dense features. Note the PCA basis here is computed imagewise. For each image, we further visualize the feature relevancy map for the category selective voxels illustrating that this method extracts the semantically relevant regions in complex compositional images.
Refer to caption: extracted/5908168/supp_stuff/fig_pdf/S5_grounding_DINO.png

Figure: Figure S.22: Grounding results using BrainSAIL for S7 DINO backbone.. We visualize the top test set images according to NSD fMRI for each category selective region. For each image, we also visualize the image-wise PCA for the distilled dense features. Note the PCA basis here is computed imagewise. For each image, we further visualize the feature relevancy map for the category selective voxels illustrating that this method extracts the semantically relevant regions in complex compositional images.
Refer to caption: extracted/5908168/supp_stuff/fig_pdf/S7_grounding_DINO.png

### A.8 Additional Visualization of Grounding using SigLIP backbone

Figure: Figure S.23: Grounding results using BrainSAIL for S1 SigLIP backbone.. We visualize the top test set images according to NSD fMRI for each category selective region. For each image, we also visualize the image-wise PCA for the distilled dense features. Note the PCA basis here is computed imagewise. For each image, we further visualize the feature relevancy map for the category selective voxels illustrating that this method extracts the semantically relevant regions in complex compositional images.
Refer to caption: extracted/5908168/supp_stuff/fig_pdf/S1_grounding_SigLIP.png

Figure: Figure S.24: Grounding results using BrainSAIL for S2 SigLIP backbone.. We visualize the top test set images according to NSD fMRI for each category selective region. For each image, we also visualize the image-wise PCA for the distilled dense features. Note the PCA basis here is computed imagewise. For each image, we further visualize the feature relevancy map for the category selective voxels illustrating that this method extracts the semantically relevant regions in complex compositional images.
Refer to caption: extracted/5908168/supp_stuff/fig_pdf/S2_grounding_SigLIP.png

Figure: Figure S.25: Grounding results using BrainSAIL for S5 SigLIP backbone.. We visualize the top test set images according to NSD fMRI for each category selective region. For each image, we also visualize the image-wise PCA for the distilled dense features. Note the PCA basis here is computed imagewise. For each image, we further visualize the feature relevancy map for the category selective voxels illustrating that this method extracts the semantically relevant regions in complex compositional images.
Refer to caption: extracted/5908168/supp_stuff/fig_pdf/S5_grounding_SigLIP.png

Figure: Figure S.26: Grounding results using BrainSAIL for S7 SigLIP backbone.. We visualize the top test set images according to NSD fMRI for each category selective region. For each image, we also visualize the image-wise PCA for the distilled dense features. Note the PCA basis here is computed imagewise. For each image, we further visualize the feature relevancy map for the category selective voxels illustrating that this method extracts the semantically relevant regions in complex compositional images.
Refer to caption: extracted/5908168/supp_stuff/fig_pdf/S7_grounding_SigLIP.png

### A.9 Implementation Details

#### Encoder training.

Our experiments utilize a mixture of GPUs including Nvidia V100 (16GB and 32GB), 2080 Ti, A6000, 6000Ada, L40S, and 4090 cards. The network training code is implemented in PyTorch. For encoder training, we employ the Adam optimizer with a decoupled weight decay of $2\times 10^{-2}$. The initial learning rate is set to $3\times 10^{-4}$ and decays exponentially to $1.5\times 10^{-4}$ over 100 epochs. Each subject is trained independently. All backbone networks are frozen, and operate in fp16 mode.

For encoder training, we always utilize the network’s native input resolution. For CLIP, we resize images to $224\times 224$. For DINO, we resize images to $518\times 518$. For SigLIP using the AM-RADIO backbone, and we resize images to $768\times 768$. After resizing, we augment the images with random pixel-wise value scaling between 0.95 and 1.05, followed by normalization using each network’s respective image mean and variance. Before network input, images are randomly offset by up to 4 pixels horizontally and vertically, with edge padding filling the resulting empty pixels. Independent Gaussian noise ($\mu=0$, $\sigma^{2}=0.05$) is added to each pixel.

#### Feature extraction.

We perform positional encoding interpolation for CLIP and DINO networks in order to extract higher resolution embeddings. For CLIP ViT-B/16 we modify the network to accept images of $896\times 896$ (4x upsampling; $56\times 56$ final patch resolution); For DINOv2 ViT-B/14+reg we modify the network to accept images of $1036\times 1036$ (2x upsampling; $74\times 74$ final patch resolution); For SigLIP based on Nvidia’s RADIOv2.5 ViT-B/16, we do not perform upsampling, and have $48\times 48$ final patch resolution. As noted in the AM-RADIO paper, the spatial patch features of these networks are highly robust to positional encoding upsampling.
We perform $51$ augmentation steps for CLIP, where the first augmentation step is null (no shift or flipping). For DINO and SigLIP, as these networks contain registers which mitigate the artifacts to a certain degree, we perform $25$ augmentation steps.

For CLIP, we utilize the NACLIP self-attention modification, with gaussian std set to $10.0$. For DINO and SigLIP, we use the MaskCLIP self-attention modification. For SigLIP specifically, since we are using the AM-RADIO implementation, we apply their provided SigLIP adapter head to the extracted features. As shown in section A.5, this enables zero-shot probing with the official SigLIP text encoder using the extracted dense features.

#### Computational cost.

Our “Learning-Free Distillation Module” is highly efficient. With pre-extracted dense features for each augmentation, we observe our procedure generally takes less than $0.2$ seconds on a GPU, this is roughly $100\times$ faster than “Denoising Vision Transformers” on the same hardware and similarly ignoring the feature extraction cost.

#### CLIP sentences.

We define a set of natural language captions to help us evaluate the alignment between fMRI region-wise relevance maps and concepts. For every caption and every image, we compute a relevance map using the CLIP text encoder. The category that contains the caption with the highest pearson correlation to the fMRI relevance map is assigned as the category.

face_class = [”A face facing the camera”, ”A photo of a face”, ”A photo of a human face”, ”A photo of faces”, ”A photo of a person’s face”, ”A person looking at the camera”, ”People looking at the camera”,”A portrait of a person”, ”A portrait photo”]
body_class = [”A photo of a torso”, ”A photo of torsos”, ”A photo of limbs”, ”A photo of bodies”, ”A photo of a person”, ”A photo of people”, ”A photo of a body”, ”A person’s arms”, ”A person’s legs”, ”A photo of hands”]
scene_class = [”A photo of a bedroom”, ”A photo of an office”,”A photo of a hallway”, ”A photo of a doorway”, ”A photo of interior design”, ”A photo of a building”, ”A photo of a house”, ”A photo of nature”, ”A photo of landscape”, ”A landscape photo”, ”A photo of trees”, ”A photo of grass”]
food_class = [”A photo of food”, ”A photo of cuisine”, ”A photo of fruit”, ”A photo of foodstuffs”, ”A photo of a meal”, ”A photo of bread”, ”A photo of rice”, ”A photo of a snack”, ”A photo of pastries”, ”A photo of vegetables”, ”A photo of pizza”, ”A photo of soup”, ”A photo of meat”, ”A photo of candy”]
text_class = [”A photo of words”, ”A photo of glyphs”, ”A photo of a glyph”, ”A photo of text”, ”A photo of numbers”, ”A photo of a letter”, ”A photo of letters”, ”A photo of writing”, ”A photo of text on an object”]

## Peer Reviews

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: Was the same adapter used in Figures 1a and 1b? Also, were the adapter parameters frozen in Figure 1b? If they were frozen, why not use clean dense features to train the adapter?

How does the quality of the dense features impact the adapter's behavior? I suggest adding an ablation study to explore this effect.
soundness: 3
strengths: This is clearly written paper and makes a clear contribution. The idea of isolating specific visual features to determine selectivity effects in different cortical areas is novel and interesting. The proposed method can be used to explore the selectivity of higher visual cortex with respect to localized scene structure and image properties. This work achieves promising open vocabulary CLIP-based segmentation results.
summary: The paper proposes a method, BrainSAIL, which aims to disentangle complex images into their semantically meaningful components and locate them to brain voxels or regions. With the prevalence of pretrained models (trained on very large amounts of data), deep learning offers a promising way to explore how semantics are organized in the brain cortex. Compared to prior methods, BrainSAIL focuses on selectivity in single-object images at the broad category level, thereby enabling a richer decomposition grounded in the full semantic complexity of natural visual experiences.
weaknesses: The current experimental comparisons (open-vocabulary segmentation) are limited, which restricts a comprehensive evaluation of the proposed method. The proposed approach appears to closely resemble BrainSCUBA. Certain technical details are unclear, which makes it challenging to fully understand or reproduce the method.

---

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: 1. My understanding is that the voxel-wise adapters predict a scalar brain response from an M-dimensional output of the backbone model. How is the voxel-wise adapter then applied to the dense features? I am thinking the dense features have a shape (H, W, M), and the adapter transforms the last dimension to get a scalar (H, W) activation map (figure 1.b). I found this a bit unclear in the paper.

2. The learning free distillation is applying random spatial transformations to the input image, applies the inverse transformation to the features, and then averages them all together. Is my understanding correct? 

3. I do not understand what equations 3-5 are about.

4. Figure 4 states that the UMAP basis is computed image-wise. I found it unclear how this was done, and what is the significance of an image-wise UMAP?
soundness: 3
strengths: The paper is well written and clear. The main contribution is the use of model features to create voxel and image-wise spatial contribution maps. This is quite useful as an interpretation technique. The use of pixel-wise metrics like depth, saturation, and luminance are a powerful extension of the method.
summary: Methods:
- Embeddings from deep neural networks are used to predict fMRI recorded brain-responses.
- Features are extracted from the model and input to a denoising process. The smoothed features are then input to the fMRI encoder (I think) to create spatial attribution maps.
- UMAP is applied to the encoder parameters, which uncovers previously reported category selective regions for faces, places, words, food, and bodies.
- The spatial attribution maps are correlated with pixel depth, color saturation, and color luminance to create voxel-wise correlation values for these image properties. Place areas were found to be more correlated with depth, food areas with color, and OPA with color/luminance.
weaknesses: All of the investigated category-selective areas (food, places, words, faces, bodies) are previously reported. Would be interesting so see this tested on less common or more fine-grained categories.

I found the explanation of the methods is a bit confusing (see questions). Could use clearer high-level descriptions before diving into the finer details.

I think there should be a visual comparison of attribution maps for different voxel categories with the same images. I was able to compare a few in figure 3 since there are some repeated images in words/food and face/body voxels. If the method is working then the word voxels should highlight words more than voxels for other categories. I would expect to see the face voxels highlighting faces a lot more than body parts, and the opposite for body voxels. However in the picture of the baby shared between the body/face voxels I don't see much of a difference in their attribution maps.

---

confidence: 4
contribution: 4
initial_rating: 8
presentation: 4
questions: - “We validate this dense feature mapping method on a large-scale fMRI dataset consisting of human participants viewing many thousands of diverse natural images that span a wide range of semantic categories and visual statistics (Allen et al.,2022).”
    - The work by Kamitani recently showed that UMAP gave something like 40 clusters of semantic diversity at a broad level, forcing us to re-evaluate our assumed understanding of just how semantically varied NSD actually is. Do you believe the findings in the "spurious reconstructions" paper have an implication on the notion of semantic diversity as described in your work?

- The introduction has repeated references to BrainSAIL’s dense embedding framework but I’m not sure why this point is continually brought up as the vision model embeddings on the natural image are dense to begin with and I can’t see that the implementation of a step to make embeddings more dense is applied anywhere, so I was a bit lost on this point. You have the adapters and a linear probe and everything always seems to be in a dense (not non-sparse) setting. Some clarification on this point would be welcome in the paper as you seem to mean something different to how I understand the same phrasing.

- Was there supposed to be a section explaining the data choices for the brain data (surface vs voxels, subject-space vs template-space) etc.? This is kind of left to be assumed from other sections of the data but I think a small section in the supplemental outlining the data usage choices more specifically would render the procedure more complete with regard to necessary experimental details. For example, the fact you’re using the masks in Jain et al. (2023) implies working in standard (normalised) template space, but I sort of expected being able to verify this explicitly via some technical description somewhere in the supplemental materials.

- You described how you selected varying functional regions from the dataset (food masks in Jain et al. (2023) and t > 2 in the Stigliani fLoc dataset) but the selected voxels being plotted, the selection of them, is not described and also a piece of missing information. Was this taken by looking at noise ceiling values calculated in NSD and if so, what value was used as the threshold?
- Lines 427-429: you identify the areas surrounding the FFA to be selective for colour, and point out this corresponds to the food area previously identified by others. Can you elaborate on the apparent contradiction between your assumption that this is actually a food area (identified earlier in the paper) or is it one driven by high colour-sensitivity and therefore might be confounded with highly colourful food images?
- Figure 7: I don’t understand how these values are calculated and the y-axis is not labelled. I wanted to ask you to be more precise on what “spatial similarity” means in terms of the method use to derive the results.
- Not quite sure the logic for why the procedure described with the learning-free distillation module results in denoised features. Why does applying a transform, running an image through an encoder, then projecting the results back again, result in denoised features? It seems to work but I just felt like the method was explained in the "how" and not "why" it works. The paper could be made stronger by adding in some intuition in the relevant section.
soundness: 4
strengths: - The authors show alternatives to high-frequency artefact removal that are more computationally efficient than other solutions out there, namely by recognising the existing common method of adding register tokens in ViTs can be replaced with their proposed learning-free distillation module, which avoids the need for (computationally demanding) additional training.
- The authors show that this method is a good way to target high confounds in naturalistic images by studying things like colour confounding with food images, luminance modulations in inside/outside scenes, depth information in varying images containing varying “reach spaces” etc.
- The paper highlights potential equivalence between representations derived under varying training algorithms and questions what is being learned and the generalisation of features. I found this to be timely as recent discussions with LLMs and the utility of methods like BrainScore have been brought into question. I think we need to be asking these questions across the board and I appreciate this final section of the paper that raises this issue.
- Visualisations are very nice and clear, making some more computationally dense text a bit more clear with reference to specific examples that underlie various findings (e.g. the effects of no language supervision in DINO and the downstream affects on the derived image maps)
summary: This paper aims to explore the semantic topography in human visual cortex using fMRI. The main goal is to disambiguate the typically confounded nature of multiple categories which is a feature/issue in naturalistic images. The trend towards naturalistic datasets in recent years necessitates more advanced methods to be able to deal with such confounds and this work takes a good step forward in the right direction. This paper then goes on to look at what regions of human visual cortex are driven by low-level image features, providing a complementary view into how the authors’ method can be used beyond studying categorical selectivity. This method provides a good solution to explore such potential confounding effects in determining what is driven by high or low level visual information.

It was a very nice paper to read and shows the authors' clear expertise and attention to detail in terms of the questions we should be asking in the community.
weaknesses: - PDF is nearly 50 pages long. While ICLR allows for 10 main pages and unlimited supplementary information, the density of what is provided is a bit on the extreme side. In light of this, I still found myself searching for implementational information that I felt was missing and could be added in a revised version (see *Questions*)
- I reformulated many of the comments I first wrote as weaknesses into more directed questions in the section below, hoping that these points might be more easily addressable and will hopefully make it into the camera-ready version such that readers have a better experience of the paper by benefiting from some of the points of clarity that I ran into while reading.

---

confidence: 4
contribution: 3
initial_rating: 6
presentation: 3
questions: DINO trained without language guidance shows high sensitivity to low-level visual features, does this hurt its performance on high-level semantic selection?
soundness: 3
strengths: 1.	BrainSAIL can obtain dense embeddings without artifacts and additional training through an efficient learning distillation module for high-throughput presentation of the visual cortex on large datasets.
2.	Compared to other studies that used simplified images, BrainSAIL can isolate specific image regions that activate different cortical areas when viewing natural scenes, providing a more complete description of how real-world visual stimuli are represented and processed.
3.	The experiments are very sufficient, especially in the appendix.
4.	This work demonstrated that an artifact-free dense feature map can be derived to explore high-throughput selectivity in the visual cortex.
summary: The authors proposed a method called BrainSAIL to deal with the problem of multiple categories co-existing in the natural images, combining the image embeddings and dense visual features and further identifying specific sub-regions of each image, which reflects the selectivity patterns in different areas of the higher visual cortex. BrainSAIL can realize the semantic attributes and positioning of related objects in complex images, which helps decompose the high-level visual representation of the human brain.
weaknesses: 1.	This method has a high model dependency (e.g., CLIP, DINO, SigLIP). While this improves efficiency, it limits flexibility to adapt feature extraction to the specific needs of neural regions, potentially constraining the granularity of semantic selectivity.
2.	The method's fMRI training uses specific datasets (e.g., NSD), which may introduce bias, limiting generalizability across different populations or visual tasks—especially when the dataset’s images or semantic information don't cover the full range of possible visual stimuli.

---

confidence: 2
contribution: 3
initial_rating: 5
presentation: 3
questions: How is it possible to verify that the method is not dominated by the foundation model representation of image rather than actually representing brain semantic representations?
soundness: 3
strengths: - The method was explained clearly
- Literature review seems exhaustive
- Claims SOTA (although I couldn't verify this)
summary: The paper introduces a method for mapping the brain activity obtained using fMRI to semantically meaningful features in image space using vision transformer backbones and a parameter-free distillation process for denoising.
weaknesses: -Abstract strats with features of BrainSAIL (first few sentences) before getting into WHAT BrainSAIL is.

-It is not clear to me after training the backbone and distillation, how much of the output result is actually driven by brain data or completely rely on features of the image learned by the frozen foundation model.

-SOTA was claimed but I wasn't able to find any comparison to other relevant methods in the paper.

Meta-review: This paper proposes a method, named BrainSAIL, which aims at investigating the semantic topology of the human brain by analyzing fMRI scans as correlated to semantic concepts in natural images. It leverages dense visual features and whole image embeddings extracted by a deep learning pre-trained model to identify specific sub-regions of each image, reflecting the selectivity patterns in different areas of the higher visual cortex.

This work was mainly appreciated since the beginning (6, 6, 8, 6, 5), lately becoming unanimously positive (6, 6, 8, 6, 6) after rebuttal.

Main significant, valuable aspects concern the clear contribution of the designed task and the approach to cope with it, the clarity of the presentation, and the principled experimentation analysis.

There are also some issues, mainly related to the novelty, some confusing parts in the method's explanation, some weak aspects of the experimental analysis, as related to comparisons and the presentation of the results, and a request of further results as considering fine-grained categories. Other punctual comments have also been raised by a couple of reviewers.

Overall, authors provided extensive and accurate comments to all raised remarks, which were appreciated by the reviewers who maintained or raise their scores.

In conclusion, this paper is recommended for acceptance to ICLR 2025.

Decision: Accept (Poster)