Brain Mapping with Dense Features: Grounding Cortical Semantic Selectivity in Natural Images With Vision Transformers

Authors: Andrew Luo, Jacob Yeung, Rushikesh Zawar, Shaurya Rajat Dewan, Margaret Marie Henderson, Leila Wehbe, Michael J. Tarr
Primary area: applications to neuroscience & cognitive science
Keywords: fMRI, visual cortex, neuroscience, cognitive science, brain, vision transformer, semantic selectivity

Abstract:
We introduce BrainSAIL (Semantic Attribution and Image Localization), a method for linking neural selectivity with spatially distributed semantic visual concepts in natural scenes. BrainSAIL leverages recent advances in large-scale artificial neural networks, using them to provide insights into the functional topology of the brain. To overcome the challenge presented by the co-occurrence of multiple categories in natural images, BrainSAIL exploits semantically consistent, dense spatial features from pre-trained vision models, building upon their demonstrated ability to robustly predict neural activity. This method derives clean, spatially dense embeddings without requiring any additional training, and employs a novel denoising process that leverages the semantic consistency of images under random augmentations. By unifying the space of whole-image embeddings and dense visual features and then applying voxel-wise encoding models to these features, we enable the identification of specific subregions of each image which drive selectivity patterns in different areas of the higher visual cortex. This provides a powerful tool for dissecting the neural mechanisms that underlie semantic visual processing for natural images. We validate BrainSAIL on cortical regions with known category selectivity, demonstrating its ability to accurately localize and disentangle selectivity to diverse visual concepts. Next, we demonstrate BrainSAIL's ability to characterize high-level visual selectivity to scene properties and low-level visual features such as depth, luminance, and saturation, providing insights into the encoding of complex visual information. Finally, we use BrainSAIL to directly compare the feature selectivity of different brain encoding models across different regions of interest in visual cortex. Our innovative method paves the way for significant advances in mapping and decomposing high-level visual representations in the human brain.

Decision: accept (poster)
Reviewer scores: 6, 6, 8, 6, 6

Reviews and discussion:
<thread>
creation date:
1730158451410

thread_type:
official_review

<thread_object>
creator: reviewer_pk4z

<content>
summary:
The paper introduces a method for mapping the brain activity obtained using fMRI to semantically meaningful features in image space using vision transformer backbones and a parameter-free distillation process for denoising.

strengths:
- The method was explained clearly
- Literature review seems exhaustive
- Claims SOTA (although I couldn't verify this)

weaknesses:
-Abstract strats with features of BrainSAIL (first few sentences) before getting into WHAT BrainSAIL is.

-It is not clear to me after training the backbone and distillation, how much of the output result is actually driven by brain data or completely rely on features of the image learned by the frozen foundation model.

-SOTA was claimed but I wasn't able to find any comparison to other relevant methods in the paper.

questions:
How is it possible to verify that the method is not dominated by the foundation model representation of image rather than actually representing brain semantic representations?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730560159348

thread_type:
official_review

<thread_object>
creator: reviewer_dukj

<content>
summary:
The authors proposed a method called BrainSAIL to deal with the problem of multiple categories co-existing in the natural images, combining the image embeddings and dense visual features and further identifying specific sub-regions of each image, which reflects the selectivity patterns in different areas of the higher visual cortex. BrainSAIL can realize the semantic attributes and positioning of related objects in complex images, which helps decompose the high-level visual representation of the human brain.

strengths:
1.	BrainSAIL can obtain dense embeddings without artifacts and additional training through an efficient learning distillation module for high-throughput presentation of the visual cortex on large datasets.
2.	Compared to other studies that used simplified images, BrainSAIL can isolate specific image regions that activate different cortical areas when viewing natural scenes, providing a more complete description of how real-world visual stimuli are represented and processed.
3.	The experiments are very sufficient, especially in the appendix.
4.	This work demonstrated that an artifact-free dense feature map can be derived to explore high-throughput selectivity in the visual cortex.

weaknesses:
1.	This method has a high model dependency (e.g., CLIP, DINO, SigLIP). While this improves efficiency, it limits flexibility to adapt feature extraction to the specific needs of neural regions, potentially constraining the granularity of semantic selectivity.
2.	The method's fMRI training uses specific datasets (e.g., NSD), which may introduce bias, limiting generalizability across different populations or visual tasks—especially when the dataset’s images or semantic information don't cover the full range of possible visual stimuli.

questions:
DINO trained without language guidance shows high sensitivity to low-level visual features, does this hurt its performance on high-level semantic selection?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730642004716

thread_type:
official_review

<thread_object>
creator: reviewer_pesm

<content>
summary:
This paper aims to explore the semantic topography in human visual cortex using fMRI. The main goal is to disambiguate the typically confounded nature of multiple categories which is a feature/issue in naturalistic images. The trend towards naturalistic datasets in recent years necessitates more advanced methods to be able to deal with such confounds and this work takes a good step forward in the right direction. This paper then goes on to look at what regions of human visual cortex are driven by low-level image features, providing a complementary view into how the authors’ method can be used beyond studying categorical selectivity. This method provides a good solution to explore such potential confounding effects in determining what is driven by high or low level visual information.

It was a very nice paper to read and shows the authors' clear expertise and attention to detail in terms of the questions we should be asking in the community.

strengths:
- The authors show alternatives to high-frequency artefact removal that are more computationally efficient than other solutions out there, namely by recognising the existing common method of adding register tokens in ViTs can be replaced with their proposed learning-free distillation module, which avoids the need for (computationally demanding) additional training.
- The authors show that this method is a good way to target high confounds in naturalistic images by studying things like colour confounding with food images, luminance modulations in inside/outside scenes, depth information in varying images containing varying “reach spaces” etc.
- The paper highlights potential equivalence between representations derived under varying training algorithms and questions what is being learned and the generalisation of features. I found this to be timely as recent discussions with LLMs and the utility of methods like BrainScore have been brought into question. I think we need to be asking these questions across the board and I appreciate this final section of the paper that raises this issue.
- Visualisations are very nice and clear, making some more computationally dense text a bit more clear with reference to specific examples that underlie various findings (e.g. the effects of no language supervision in DINO and the downstream affects on the derived image maps)

weaknesses:
- PDF is nearly 50 pages long. While ICLR allows for 10 main pages and unlimited supplementary information, the density of what is provided is a bit on the extreme side. In light of this, I still found myself searching for implementational information that I felt was missing and could be added in a revised version (see *Questions*)
- I reformulated many of the comments I first wrote as weaknesses into more directed questions in the section below, hoping that these points might be more easily addressable and will hopefully make it into the camera-ready version such that readers have a better experience of the paper by benefiting from some of the points of clarity that I ran into while reading.

questions:
- “We validate this dense feature mapping method on a large-scale fMRI dataset consisting of human participants viewing many thousands of diverse natural images that span a wide range of semantic categories and visual statistics (Allen et al.,2022).”
    - The work by Kamitani recently showed that UMAP gave something like 40 clusters of semantic diversity at a broad level, forcing us to re-evaluate our assumed understanding of just how semantically varied NSD actually is. Do you believe the findings in the "spurious reconstructions" paper have an implication on the notion of semantic diversity as described in your work?

- The introduction has repeated references to BrainSAIL’s dense embedding framework but I’m not sure why this point is continually brought up as the vision model embeddings on the natural image are dense to begin with and I can’t see that the implementation of a step to make embeddings more dense is applied anywhere, so I was a bit lost on this point. You have the adapters and a linear probe and everything always seems to be in a dense (not non-sparse) setting. Some clarification on this point would be welcome in the paper as you seem to mean something different to how I understand the same phrasing.

- Was there supposed to be a section explaining the data choices for the brain data (surface vs voxels, subject-space vs template-space) etc.? This is kind of left to be assumed from other sections of the data but I think a small section in the supplemental outlining the data usage choices more specifically would render the procedure more complete with regard to necessary experimental details. For example, the fact you’re using the masks in Jain et al. (2023) implies working in standard (normalised) template space, but I sort of expected being able to verify this explicitly via some technical description somewhere in the supplemental materials.

- You described how you selected varying functional regions from the dataset (food masks in Jain et al. (2023) and t > 2 in the Stigliani fLoc dataset) but the selected voxels being plotted, the selection of them, is not described and also a piece of missing information. Was this taken by looking at noise ceiling values calculated in NSD and if so, what value was used as the threshold?
- Lines 427-429: you identify the areas surrounding the FFA to be selective for colour, and point out this corresponds to the food area previously identified by others. Can you elaborate on the apparent contradiction between your assumption that this is actually a food area (identified earlier in the paper) or is it one driven by high colour-sensitivity and therefore might be confounded with highly colourful food images?
- Figure 7: I don’t understand how these values are calculated and the y-axis is not labelled. I wanted to ask you to be more precise on what “spatial similarity” means in terms of the method use to derive the results.
- Not quite sure the logic for why the procedure described with the learning-free distillation module results in denoised features. Why does applying a transform, running an image through an encoder, then projecting the results back again, result in denoised features? It seems to work but I just felt like the method was explained in the "how" and not "why" it works. The paper could be made stronger by adding in some intuition in the relevant section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730690057011

thread_type:
official_review

<thread_object>
creator: reviewer_yym7

<content>
summary:
Methods:
- Embeddings from deep neural networks are used to predict fMRI recorded brain-responses.
- Features are extracted from the model and input to a denoising process. The smoothed features are then input to the fMRI encoder (I think) to create spatial attribution maps.
- UMAP is applied to the encoder parameters, which uncovers previously reported category selective regions for faces, places, words, food, and bodies.
- The spatial attribution maps are correlated with pixel depth, color saturation, and color luminance to create voxel-wise correlation values for these image properties. Place areas were found to be more correlated with depth, food areas with color, and OPA with color/luminance.

strengths:
The paper is well written and clear. The main contribution is the use of model features to create voxel and image-wise spatial contribution maps. This is quite useful as an interpretation technique. The use of pixel-wise metrics like depth, saturation, and luminance are a powerful extension of the method.

weaknesses:
All of the investigated category-selective areas (food, places, words, faces, bodies) are previously reported. Would be interesting so see this tested on less common or more fine-grained categories.

I found the explanation of the methods is a bit confusing (see questions). Could use clearer high-level descriptions before diving into the finer details.

I think there should be a visual comparison of attribution maps for different voxel categories with the same images. I was able to compare a few in figure 3 since there are some repeated images in words/food and face/body voxels. If the method is working then the word voxels should highlight words more than voxels for other categories. I would expect to see the face voxels highlighting faces a lot more than body parts, and the opposite for body voxels. However in the picture of the baby shared between the body/face voxels I don't see much of a difference in their attribution maps.

questions:
1. My understanding is that the voxel-wise adapters predict a scalar brain response from an M-dimensional output of the backbone model. How is the voxel-wise adapter then applied to the dense features? I am thinking the dense features have a shape (H, W, M), and the adapter transforms the last dimension to get a scalar (H, W) activation map (figure 1.b). I found this a bit unclear in the paper.

2. The learning free distillation is applying random spatial transformations to the input image, applies the inverse transformation to the features, and then averages them all together. Is my understanding correct? 

3. I do not understand what equations 3-5 are about.

4. Figure 4 states that the UMAP basis is computed image-wise. I found it unclear how this was done, and what is the significance of an image-wise UMAP?
</content>
</thread_object>

</thread>
<thread>
creation date:
1731154367919

thread_type:
official_review

<thread_object>
creator: reviewer_pzhq

<content>
summary:
The paper proposes a method, BrainSAIL, which aims to disentangle complex images into their semantically meaningful components and locate them to brain voxels or regions. With the prevalence of pretrained models (trained on very large amounts of data), deep learning offers a promising way to explore how semantics are organized in the brain cortex. Compared to prior methods, BrainSAIL focuses on selectivity in single-object images at the broad category level, thereby enabling a richer decomposition grounded in the full semantic complexity of natural visual experiences.

strengths:
This is clearly written paper and makes a clear contribution. The idea of isolating specific visual features to determine selectivity effects in different cortical areas is novel and interesting. The proposed method can be used to explore the selectivity of higher visual cortex with respect to localized scene structure and image properties. This work achieves promising open vocabulary CLIP-based segmentation results.

weaknesses:
The current experimental comparisons (open-vocabulary segmentation) are limited, which restricts a comprehensive evaluation of the proposed method. The proposed approach appears to closely resemble BrainSCUBA. Certain technical details are unclear, which makes it challenging to fully understand or reproduce the method.

questions:
Was the same adapter used in Figures 1a and 1b? Also, were the adapter parameters frozen in Figure 1b? If they were frozen, why not use clean dense features to train the adapter?

How does the quality of the dense features impact the adapter's behavior? I suggest adding an ablation study to explore this effect.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732213267340

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General Response

comment:
We are grateful to all reviewers for their constructive suggestions, which we agree will improve the clarity and communication of our work.

We are very encouraged by reviewers’ evaluation on the quality of this paper. All five reviewers find the work interesting -- "novel and interesting" (**PZHq**); "quite useful as an interpretation technique" (**yyM7**); "highlights potential equivalence between representations" (**PeSm**); provides "a more complete description of how real-world visual stimuli are represented and processed" (**dUkJ**); "method was explained clearly" (**PK4z**). 

## General clarifications
### 1.1 Scope and experiments
* We propose BrainSAIL -- A method to localize regions in a image that a brain area is selective to. These spatial attribution maps utilize the dense embeddings from vision transformers.
* We leverage modifications to the self-attention mechanism that ensure the dense embeddings lie in the same embedding space as the summary token. We further denoise this dense representation using *learning-free distillation*. Inspired by multiview consistency in 3D neural field learning, we propose to generate 2D "views" with image augmentations.
* To mitigate the issue of slow neural field learning, we show mathematically that mean-aggregation achieves the same optimal representation as gradient-based MSE optimization. We find that our method consistently improves mIoU and correlation scores over NACLIP in **Table 1**, a method which claimed state of the art open-vocabulary segmentation results earlier this year. 

We evaluate our method in four different ways, and leverage the ground truth category selectivity of voxels, measured using a functional localizer as part of that evaluation. We compare models that achieve similar prediction performance (CLIP, DINO, SigLIP), and find that they differ in terms of the features used. 

* First, we perform dimensionality reduction on the higher visual cortex, and apply the same basis to image dense features. 
    * We use UMAP applied to fMRI encoder weights and dense CLIP/DINO/SigLIP features.
    * We find that the image components align with known functionality in the visual cortex, including for people (EBA/FFA), food, and scenes (RSC/OPA/PPA)
* Second, we perform spatial attribution for regions of known functional selectivity.
    * We visualize the attributions, and find that they can function well even with brain regions selective to visually diverse objects like food. 
    * We quantify the attributions by computing the category with the best spatial attribution alignment using the CLIP-text encoder for each brain region, and find that our method can always identify the correct top-category for face/body/place/food-selective voxels, but do not always agree in the word region (likely due to food and word region overlap in the brain). 
* Third, we investigate low-level voxel-wise visual correlates using voxel-wise BrainSAIL. Across the visual cortex using depth, color saturation, and color luminance.
    * We find that depth correlates can identify scene regions.
    * We find the region surrounding FFA commonly identified to be food selective to be correlated with high color saturation
    * We find that OPA is divided into high/low color luminance, corresponding with previously identified outdoor/indoor selectivity.
* Finally, we compare the features utilized by different brain encoders. We investigate CLIP (vision-language contrastive), DINO (vision contrastive), and SigLIP (vision-language non-contrastive) models. While they achieve similar $R^2$, the models differ in terms of the features used for brain prediction.
    * We find that models that utilize language are more robust to visually diverse features that are part of the same object.
    * Vision only models are more sensitive and brittle to components that are visually different in color or shading.
    * Models differ the most for visually diverse objects (words and food), while agreeing on images which could be characterized by simple features (rectilinear features in scenes)

## 2. Updates and new experiments in the revision
1. We have clarified that our voxel-wise weights are frozen when used for inference on the dense features. (Page 3, **PZHq**)
2. We have added a discussion and citation on the potential limitations caused by using NSD (Page 10, **PeSm**, **dUkJ**)
4. We have added an experiment comparing the segmentation capabilities of CLIP and the brain (Page 20, **yyM7**, **PK4z**)
6. We have reorganized the supplemental and added additional information on fMRI data processing (Page 19, **PeSm**)
7. We better motivate our learning-free distillation, and clarify the purpose of equations 3 to 5. (Page 19, **yyM7**, **PeSm**)
8. We have added a previously missing axis label (Page 10, **PeSM**)

We genuinely appreciate the suggestions, and believe our paper will be improved with your feedback. Please let us know if you have any additional questions or comments!
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper proposes a method, named BrainSAIL, which aims at investigating the semantic topology of the human brain by analyzing fMRI scans as correlated to semantic concepts in natural images. It leverages dense visual features and whole image embeddings extracted by a deep learning pre-trained model to identify specific sub-regions of each image, reflecting the selectivity patterns in different areas of the higher visual cortex.

This work was mainly appreciated since the beginning (6, 6, 8, 6, 5), lately becoming unanimously positive (6, 6, 8, 6, 6) after rebuttal.

Main significant, valuable aspects concern the clear contribution of the designed task and the approach to cope with it, the clarity of the presentation, and the principled experimentation analysis.

There are also some issues, mainly related to the novelty, some confusing parts in the method's explanation, some weak aspects of the experimental analysis, as related to comparisons and the presentation of the results, and a request of further results as considering fine-grained categories. Other punctual comments have also been raised by a couple of reviewers.

Overall, authors provided extensive and accurate comments to all raised remarks, which were appreciated by the reviewers who maintained or raise their scores.

In conclusion, this paper is recommended for acceptance to ICLR 2025.