Finding Shared Decodable Concepts and their Negations in the Brain

## Abstract

Abstract Prior work has offered evidence for functional localization in the brain; different anatomical regions preferentially activate for certain types of visual input. For example, the fusiform face area preferentially activates for visual stimuli that include a face. However, the spectrum of visual semantics is extensive, and only a few semantically-tuned patches of cortex have so far been identified in the human brain. Using a multimodal (natural language and image) neural network architecture (CLIP, Radford et al. ( 2021 ) , we train a highly accurate contrastive model that maps brain responses during naturalistic image viewing to CLIP embeddings. We then use a novel adaptation of the DBSCAN clustering algorithm to cluster the parameters of these participant-specific contrastive models. This reveals what we call Shared Decodable Concepts (SDCs): clusters in CLIP space that are decodable from common sets of voxels across multiple participants. Examining the images most and least associated with each SDC cluster gives us additional insight into the semantic properties of each SDC. We note SDCs for previously reported visual features (e.g. orientation tuning in early visual cortex) as well as visual semantic concepts such as faces, places and bodies. In cases where our method finds multiple clusters for a visuo-semantic concept, the least associated images allow us to dissociate between confounding factors. For example, we discovered two clusters of food images, one driven by color, the other by shape. We also uncover previously unreported areas with visuo-semantic sensitivity such as regions of extrastriate body area (EBA) tuned for legs/hands and sensitivity to numerosity in right intraparietal sulcus, sensitivity associated with visual perspective (close/far) and more. Thus, our contrastive-learning methodology better characterizes new and existing visuo-semantic representations in the brain by leveraging multimodal neural network representations and a novel adaptation of clustering algorithms.

## 1 Introduction

The study of the visual system has revealed a significant amount of knowledge about how the brain processes and organizes visual information. The early visual system has been well characterized for its ability to extract low level visual features (e.g. lines of different orientations). However, the picture becomes more murky as one moves beyond the early visual system. There is evidence for functional localization wherein large patches of cortex activate for specific visuo-semantic properties: areas of the brain preferentially activate for visual stimuli containing faces (Kanwisher et al., ) and places (Epstein and Kanwisher, ). There have been multiple recent attempts to identify new functionally localized regions of cortex with some converging evidence; for example, a food area near PPA has been identified (Khosla et al., ). So, while there has been progress towards parcelating the functional regions of the brain, only a few such broad visuo-semantic categories have been identified and they typically cover fairly large areas of cortex. This leaves open the possibility that we may be able to detect new visuo-semantic categories in the brain or refine existing categories, and that these categories or sub-categories may become more apparent with a data-driven approach that can combine data across multiple participants.

But combining brain responses across participants is not simple. Prior research typically focused on mapping to a shared latent space (e.g. during stimulus reconstruction) for which spatial localization in the brain is not required. In cases where spatial information is required, analyses are typically applied to data which has been warped to a common template brain, often losing the connection to the unique structure of each participant’s cortical layout. We present a novel method that finds semantically meaningful shared voxel clusters in each participant’s native brain space, allowing us to sidestep the voxel alignment problem. Derived voxel clusters can be mapped to a shared space for visualization after identification of shared semantic clusters.

Our data-driven analysis of fMRI recordings during image-viewing allows not only for identification of the images that most drive neural activity in a given area, but also of images which are most associated with below-average BOLD response. This provides stronger evidence for the possible function of cortical areas, in some cases refining the hypotheses one might draw from a set of images. Brain areas showing suppressed activity in response to a stimuli is consistent with recently reported “offsembles” of neurons (Pérez-Ortega_Akrouh_Yuste_2024) which show inhibited activity in response to e.g. visual stimuli, implying that, when interpreting neural representations, suppression of firing may be as important as firing.

We identify semantic concepts that drive activity in patches of cortex across multiple participants and find that their localization is fairly consistent across participants. To accomplish this, we introduce a novel variant of DBSCAN that combines data from multiple fMRI participants, identifying the shared decodable concepts (SDC) by specifically searching for SDCs present in multiple subjects.

## 2 Decoding CLIP-Space from Brain Images

To identify shared decodable concepts (SDCs) in the brain, we derived a mapping from anatomical brain space to a representational space in which we can explore semantic sensitivity to visual input. In this section we describe the components of this mapping: a dataset of fMRI recordings (NSD), a multimodal image-text embedding model (CLIP), and our method to map from per-image brain responses to their associated multimodal embeddings. We consider two models in this section, and verify that our proposed contrastive decoder outperforms a baseline ridge regression model.

### 2.1 fMRI Data

The natural scenes dataset (NSD) is a massive fMRI dataset acquired to study the underpinnings of natural human vision (Allen et al., ). Eight participants were presented with 30,000 images (10,000 unique images over 3 repetitions) from the Common Objects in Context (COCO) naturalistic image dataset (Lin et al., ). A set of 1,000 shared images were shown to all participants, while the other 9,000 images were unique to each participant. Single-trial beta weights were derived from the fMRI time series using the GLMSingle toolbox (Prince et al., ). This method fits numerous haemodynamic response functions (HRFs) to each voxel, as well as an optimized denoising technique and voxelwise fractional ridge regression, specifically for single-trial fMRI acquisition paradigms. Some participants did not complete all fMRI recording sessions and three sessions were held out by the NSD team for the Algonauts challenge. Further details can be found in Allen et al. ().

### 2.2 Representational Space for Visual Stimuli

To generate representations for each stimulus image, we use CLIP (Radford et al., ), a model trained on over 400 million text-image pairs with a contrastive language-image pretraining objective. CLIP consists of a text-encoder and image-encoder that jointly learns a shared low-dimensional space trained to maximize the cosine similarity of corresponding text and image embeddings. We use the 32-bit Transformer model (ViT-B/32) implementation of CLIP to create a 512-dimensional representation for each stimulus image used in the NSD experiment. We train a decoder to map from fMRI responses during image viewing to the associated CLIP vector for that same image. Notably, because CLIP is a joint image-language model, these CLIP vectors also correspond to text captions in the pretraining stage, which can be used to describe the images presented to the participants.

### 2.3 Data Preparation

#### Data Split

We split the per-image brain responses $\bm{X}$ and CLIP embeddings $\bm{Y}$ into training $(\bm{X}_{\text{Train}},\bm{Y}_{\text{Train}})$, validation $(\bm{X}_{\text{Val}},\bm{Y}_{\text{Val}})$, and test $(\bm{X}_{\text{Test}},\bm{Y}_{\text{Test}})$ folds for each of the 8 NSD participants. For each participant, the validation and test folds were chosen to have exactly 1,000 images with three presentations. (^1^11Some participants in the NSD did not complete all scanning sessions and only viewed certain images once or twice. These images are assigned to the training set, which varies in size across participants.) Of the shared 1,000 images that all participants saw, 413 images were shown three times to every participant across the sessions released by NSD. These 413 images appear in each participant’s testing fold.

#### Voxel Selection

The NSD fMRI data comes with voxelwise noise ceiling (NC) estimates that can be used to identify reliable voxels. However, the NC is calculated using the *full dataset*. Therefore, these NC estimates should not be used to extract a subset of voxels for decoding analyses because they are calculated using images in the test set, which is a form of double-dipping (Kriegeskorte et al., ). We therefore re-calculated the per-voxel noise ceiling estimates specifically on our designated training data only. We selected voxels with noise ceiling estimates above $8\%$ variance explainable to use as inputs to the decoding model. This resulted in 6k-19k voxel subsets per participant (see supplementary section 6.1 for exact per-participant voxel dimensions input into the decoder) which were normalized according to section 6.5. In our visualizations, regions of the flattened brain surface in black represent voxels that have passed this voxel selection threshold.

### 2.4 Decoding Methodology

#### Decoding Model

The decoding model $\bm{g}(\bm{X};\bm{\theta})=\bm{\hat{Y}}$ is a linear model trained to map brain responses $\bm{X}=[\bm{x}_{1},\ldots,\bm{x}_{n}],\bm{x}_{i}\in\mathbb{R}^{v}$ to brain embeddings $\bm{\hat{Y}}=[\bm{y}_{1},\ldots,\bm{y}_{n}],\bm{y}_{i}\in\mathbb{R}^{512}$. Here $n$ is the number of training instances, and $v$ is the participant-wise number of voxels that pass the noise ceiling threshold. We optimize the brain decoder using the InfoNCE definition of contrastive loss (van den Oord et al., ), which is defined below in Equations 1 and 2.

$$ $\text{Contrast}(\bm{A},\bm{B})=-\frac{1}{M}\sum_{i=1}^{M}log\bigg{(}\frac{exp( \bm{a}_{i}\cdot\bm{b}_{i}/\tau)}{\sum_{j=1}^{M}exp(\bm{a}_{i}\cdot\bm{b}_{j}/ \tau)}\bigg{)}$ (1) $$

$$ $\mathcal{L}_{\text{InfoNCE}}(\bm{A},\bm{B})=\frac{1}{2}[\text{Contrast}(\bm{A} ,\bm{B})+\text{Contrast}(\bm{B},\bm{A})]$ (2) $$

In this definition $\bm{A}=[\bm{a}_{1},\ldots,\bm{a}_{n}]$ and $\bm{B}=[\bm{b}_{1},\ldots,\bm{b}_{n}]$ are embeddings for two modalities representing the same data points, the $\cdot$ operator represents cosine similarity, and $\tau$ is a temperature hyper-parameter. The loss is minimized when the distance between matching embeddings is small, and the distance between mismatched embeddings is large. In the original CLIP setting, $\bm{A}$ and $\bm{B}$ represent embeddings for images and corresponding text captions. In our implementation, we apply the contrastive loss to image embeddings computed by a pretrained frozen CLIP model and CLIP embeddings predicted from fMRI, i.e. we optimize $\min_{\bm{\theta}}\>\mathcal{L}(\bm{\hat{Y}},\bm{Y}_{\text{CLIP}})=\mathcal{L}
(g(\bm{X};\bm{\theta}),\bm{Y})$. An illustration of the decoding procedure is given in Figure 11 (additional details on the decoding procedure are given in Section 6.1).

#### Evaluation

We evaluate our models using top-k accuracy. Figure 12 shows the results of this evaluation. The contrastive decoder outperforms ridge regression across all values of $k$. Recall that our end goal is to identify shared decodable concepts (SDC) in the brain. Our methodology for this task relies on the *predicted* CLIP vectors, and so our subsequent analyses require an accurate decoding model.

## 3 Finding Shared Brain-Decodable Concepts and their Representative Images

In this section, we describe our methods for finding and interpreting areas of the brain that respond to semantically similar images, and that are consistent across participants. We approach this problem by analyzing the weight matrices $\bm{W}^{(k)}$ from the optimized linear decoders described in section 3.4 $g^{(k)}(\bm{x}^{(k)})=\bm{W}^{(k)}\bm{x}^{(k)}=\bm{y}$ for all subjects $k\in\{1\ldots 8\}$. First, we observe that the decoder’s linear transformation $\bm{W}^{(k)}\bm{x}^{(k)}=\sum_{i=1}^{v}\bm{w}_{i}^{(k)}\cdot{x_{i}}^{(k)}$ is simply a summation of parameter vectors $\bm{w}_{i}^{(k)}\in\mathbb{R}^{512}$ that are scaled by brain response values $x_{i}^{(k)}\in\mathbb{R}$. This means that when $x_{i}^{(k)}$ is above or below baseline activation, the CLIP dimension represented by $\bm{w}_{i}^{(k)}$ is increased or decreased in the brain-decoded embedding. This motivates the key to our analysis: we view the parameter vector $\bm{w}_{i}^{(k)}$ as a CLIP vector that represents a brain-decodable concept for voxel $i$. This allows us to use cosine distance $d(\bm{w}_{i}^{(k)},\bm{w}_{j}^{(r)})$ between the weight vectors for voxels $i,j$ from participants $k,r$ as a measure of similarity between the decodable concepts of brain voxels across participants. Our objective is to apply a clustering algorithm using this metric in order to find areas in the brains of multiple participants that have a high conceptual similarity. We refer to these underlying concepts as shared decodable concepts (SDCs). We interpret the SDCs by retrieving a set of representative images associated with the brain-decoded CLIP vectors that are closest to the centroid of each SDC cluster. A schematic of the algorithm we apply is given in Figure 1.

Figure: Figure 1: Deriving SDC clusters. (a): The participant-specific linear decoders derived in Figure 6.1. (b): Our modified DBSCAN clustering procedure is applied to the linear decoders (See Figure 2 for details). (c): Our DBSCAN procedure derives binary masks over the voxels in the linear decoders for a specified number of clusters (of which one is highlighted in orange) (d): The rows corresponding to the selected voxels in the binary masks are extracted from the linear decoder matrices. (e): The 512-dimensional representations from the previous step are averaged over voxels and participants to derive a cluster centroid for each cluster derived from DBSCAN. We visualize the cluster centroid for the first DBSCAN cluster. (f): The linear decoders from (a) are applied to held-out fMRI data per-participant. Each participant saw images multiple times, so the matrix of predicted CLIP embeddings $\hat{Y}^{k}_{Test}$ is averaged over these repetitions and all linear decoding matrices are stacked (across participants) to give $\hat{Y}^{AVG}_{Test}$. (g): Cosine distance is calculated between the cluster centroids (e) and the brain-derived CLIP embeddings (f). (h): The images most associated with the cluster centroids (positive images) and most negatively associated with the cluster centroids (negative images) are identified. Positive / negative images for the SDC cluster pictured here appears to correspond to global vertical/horizontal orientation in the associated images. (i): Color-coded participant-specific voxel clusters are displayed on a flatmap of the brain’s cortical surface in common fsaverage space (overlapping areas are displayed in white). Regions of interest labels are highlighted on the flatmap image in white outlines. For the specified cluster (e), whose positive / negative images are associated with orientation, the flatmap indicates bilateral shared voxel clusters in early visual cortex.
Refer to caption: extracted/5892400/main_fig.png

### 3.1 Cross Participant Clustering

To discover shared brain-decodable concepts we apply a novel clustering method to the per-voxel model parameter vectors $\bm{w}_{i}^{(k)}$ across all participants. We base our clustering method on the density-based spatial clustering of applications with noise (DBSCAN) (Ester et al., ) algorithm, which is parameterized by a neighborhood size $\varepsilon$ and a point threshold minPts. The algorithm can be summarized using the following steps: (1) Points that have at least minPts points in their $\varepsilon$-neighborhood are marked as *core* points; (2) construct a graph $\mathcal{G}$ where the vertices are core points, and there are edges between core points that have a distance less than $\varepsilon$; (3) clusters are formed by finding the connected components of $\mathcal{G}$ and (4) the remaining *non-core* points are added to clusters if they are within the $\varepsilon$-neighborhood of a core point, otherwise, they are marked as outliers not belonging to any cluster.

In our modified cross-participant DBSCAN, we redefine the core point threshold minPts as minNeighbors. Then, in step (1) a point $\bm{w}_{i}^{(k)}$ is a core point if there are points from at least minNeighbors other participants within its $\varepsilon$ neighborhood. With this modification, a core point now identifies voxels that represent a brain-decodable concept that is shared with at least minNeighbors participants. Since there are 8 participants in the study, the maximum value of minNeighbors is 7. Steps (2) to (4) proceed as above.

Our second modification is to apply a within-participant expansion of clusters. We define a new hyperparameter $\varepsilon_{\text{expansion}}$ and apply a final expansion step: (5) All points inside the $\varepsilon_{\text{expansion}}$ neighborhood of a point in a cluster become members of that cluster with the constraint that they must be from the *same* participant. The addition of this step was motivated by our early experiments, where we noticed that there were sometimes only one or two voxels belonging to certain participants for each cluster. This allows the cluster boundaries to grow slightly larger for each participant, and we found that this did not significantly change the cluster location or semantic interpretations. Figure 2 outlines the application of our modified DBSCAN algorithm to modeling fMRI data.

We present results using a fixed value for $\texttt{minNeighbors}=3$. This means that each cluster must include at least 4 out of 8 participants in the study. This allows for the discovery of SDCs that may show up less reliably in some participants. Furthermore, we use a range of values for $\varepsilon\in\{0.5,0.55,0.6,0.65\}$ to explore clusters at different density scales. We set the expansion neighborhood size $\varepsilon_{\text{expansion}}=\text{min}(\varepsilon+0.05,0.65)$ so that the cluster boundaries can grow slightly larger than the baseline neighborhood size $\varepsilon$, but not exceeding 0.65 as we noticed neighborhoods become over-connected as $\varepsilon$ approaches 0.7. The number of cross-participant clusters found was 9, 14, 12, 11 for $\varepsilon$ values of 0.5, 0.55, 0.6, 0.65 respectively.

To reduce the impact of random initialization and combat the noise inherent in fMRI, we re-trained the decoder model 50 times for every participant and compute the average of the resulting parameter vectors. In other words, each averaged parameter vector $\bm{w}_{i}^{(k)}$ used in our analysis represents the average of 50 concepts that a voxel could represent depending on the random initialization of the decoder. We found that this significantly improved the quality and consistency of the clusters revealed by our modified DBSCAN algorithm.

Figure: Figure 2: Illustration of our DBSCAN variant applied to multi-participant fMRI data. There are 3 participants in this example and $\texttt{minNieghbors}=2$. (a): A zoomed-in view of a cluster with three core points. The outer ring around each point shows its $\varepsilon$-neighborhood and whether it is a core, border, or outlier point. Black arrows emphasize points that are neighbors. The points with neighboring points from at least 2 other distinct participants are marked as core points. Non-core points that neighbor core points are added to the cluster as border points. The remaining points are marked as outliers. (b): A zoomed-out sketch of a set of points that form 3 clusters. Since $\texttt{minNieghbors}=2$, a high-density region will form clusters if and only if it contains points from at least 3 participants.
Refer to caption: extracted/5892400/dbscan_v2.png

### 3.2 Selecting Representative Images

To interpret the semantic meaning of an SDC cluster, we first computed the cluster centroid by taking the mean of all parameter vectors within the cluster $\bm{\bar{w}}=\frac{1}{|I|}\sum_{(i,k)\in I}\bm{w}_{i}^{(k)}$ where $I$ is a set of tuples identifying the voxel and subject indices for an SDC cluster. When the cluster voxels are above or below baseline activation, the CLIP dimension represented by $\bm{\bar{w}}$ is increased or decreased in the brain-decoded embedding, respectively. We note that DBSCAN has the ability to create non-globular clusters for which the centroid might not be a good representative. In our experience the centroid is within $\varepsilon$ of a core point within the cluster, implying that it is likely fairly representative of the cluster.

In order to find the representative images for an SDC cluster, we can consider the nearby image-embeddings to the cluster centroid $\bm{\bar{w}}$. Instead of using the original CLIP embeddings for each image, we instead used the subject-specific *brain-decoded* embeddings $\bm{\hat{Y}}_{\text{Test}}^{(k)}$. This allows us to focus specifically on the information that can be decoded from fMRI, which could be a subset of the information represented by CLIP space. For images that were viewed more than once we average the embeddings within and across participants. These averaged brain-decoded embeddings from held-out test data are denoted by $\bm{\hat{Y}}_{\text{Test}}^{\text{AVG}}$. To select nearby images we define $D(\bm{w},\bm{Y})=\{d(\bm{w},\bm{y})|\bm{y}\in\bm{Y}\}$ which is the set of distances between a CLIP vector $\bm{w}$ to a set of CLIP embeddings $\bm{Y}$. We take the cosine distances between the SDC cluster centroid $D(\bm{\bar{w}},\bm{\hat{Y}}_{\text{Test}}^{\text{AVG}})$ and the *negated* centroid $D(-\bm{\bar{w}},\bm{\hat{Y}}_{\text{Test}}^{\text{AVG}})$ and retrieve the images corresponding to the smallest values in these sets as the positive and negative representative images respectively.

## 4 Results

The clustering method described in chapter 3 not only identifies previously discovered functional areas but additionally identifies new ones, both of which we explore in this section. We emphasize that there is no constraint that voxels within a shared visual semantic cluster be spatially adjacent in the brain, yet during visualization we consistently find contiguous patches both within and across participants. We can localize these patches of voxels to specific regions of interest (ROIs) both functionally and anatomically (See Section 6.3 for further details).

### 4.1 Faces

One of the first reported functionally localized areas for higher-order vision was the fusiform face area (FFA)  (Kanwisher et al., ). Our method identifies a face-related concept (Figure 4.1 $\varepsilon=0.55$, cluster 7) that is localized to FFA and includes voxels from all 8 participants. This cluster also has some voxels in the extrastriate body area (EBA), likely because many face images include part or all of a person’s body. We note that the positive representative images are not exclusively human and include a range of animal faces. The images often depict people eating and handling food, or holding other objects such as toothbrushes or cell phones.

Interestingly, the negative representative images often display bodies, but there is a striking *lack* of clearly visible faces. There are several examples where people are visible, but their faces are obscured or they are facing away from the camera. This suggests that there may be a dip in FFA activity for images where a person’s face might be expected (and thus FFA is primed to activate) but not clearly visible. (^2^22Thus the brain’s representation for the opposite of a face is a scene with the conspicuous absence of faces.)

Figure: Figure 3: Cluster 7 ($\varepsilon=0.55$). Positive images are strongly associated with faces, while the negative images represent depictions of people whose faces are not visible. Voxel clusters are primarily found in bilateral FFA and EBA.
Refer to caption: extracted/5892400/faces_epsilon0.55_cluster7.png

### 4.2 Food and Color

Previous work has specifically noted the correlation of very colorful images with food-related images, and attempted to define food areas in the absence of color  (Jain et al., ). Our method also identifies a large possibly food-related cluster that spans FFA, PPA, and V4 (cluster 0, $\varepsilon=0.55$, Figure 4). At first inspection most of the positive representative images are food-related. However, we also observed many vibrant and colorful positive images that contained no food. Strikingly, the negative representative images are *entirely gray-scale*, providing very strong evidence that this cluster may be related to color. Thus, we speculate that this cluster is not specifically related to the identification of food, but might rather correspond to any colorful image. We discuss two other food-related clusters in supplementary section 6.8.

### 4.3 Bodies

Our method identifies five notable body-related areas at $\varepsilon=0.55$ in and around EBA (downing_2001). The positive representative images for cluster 2 (Figure 5) show people and animals outside with an emphasis on legs and active movement. The negative images typically depict people indoors sitting with their legs obscured by tables. Cluster 11 (Figure 6) has a similar emphasis on hands instead of legs. People are displayed in a variety of contexts with their hands clearly visible, while the negative images are exclusively non-primate animals without hands. This cluster also shows indoor/outdoor contrast in the representative image groups and so there is some PPA activation. In Section 6.7 we outline further body-related shared voxel clusters associated with motion and a distinction between single individuals and groups of people in a sub-region of EBA.

Figure: Figure 5: Cluster 2 ($\varepsilon=0.55$). Positive images are strongly associated with presence of legs, while negative images are typically people at tables whose legs are obscured. Voxel clusters are primarily in bilateral EBA.
Refer to caption: extracted/5892400/legs_epsilon0.55_cluster2.png

### 4.4 Horizontal/Vertical Cluster

Very early work on the visual system discovered the tuning of the early visual system for lines of a particular orientation. Two clusters emerge that reflect this tuning, both at $\varepsilon=0.65$. Cluster 1 (Figure 7) has a strong horizontal component with strong horizon lines or large objects spanning the middle of the visual field creating a horizon-like line. Notably, the negative images are images with a strong vertical component. The inverse is true for cluster 7 (Figure 8) which shows strong verticality in the positive images, and horizontal in the negative. Interestingly some of the negative images from cluster 1 appear as positive images in cluster 7, and vice versa. The localization of both clusters is V2/V3.

Figure: Figure 7: Cluster 1 ($\varepsilon=0.65$). Positive images are associated with a horizontal mid-line, while negative images display a vertical mid-line. Voxel clusters are primarily in right hemisphere V2.
Refer to caption: extracted/5892400/horizontal_epsilon0.65_cluster1.png

### 4.5 Repeated Elements (Numerosity) Cluster

We observed that cluster 12 ($\varepsilon=0.55$, Figure 9) has positively associated images typically with repeated elements of related items, while negative images typically contained singular instances of items. We suggest that this SDC could be related to quantity processing and numerosity. We observed voxel clusters primarily in right OPA and IPS regions. As with other clusters, naturalistic images often implicates functional areas associated with place (OPA). The IPS has been widely reported in prior work to be associated with quantity processing, e.g. grammatical number processing Carreiras et al. (), mathematical processing deficits Ganor-Stern et al. () and numerical processing Koch et al. ()). Our observations in the visual domain provide converging evidence that the right IPS is associated with quantity processing across multiple modalities.

Figure: Figure 9: Cluster 12 ($\varepsilon=0.55$). Positive images are associated with repeated elements, while negative images are single objects or individuals. Voxel clusters are primarily located in OPA and IPS.
Refer to caption: extracted/5892400/repetition_epsilon0.55_cluster12.png

### 4.6 Close vs Far Cluster

With $\varepsilon=0.65$, cluster 2 (Figure 10) seems to represent big things (airplanes, busses) and close up pictures of larger animals (zebras, elephants). The negatively associated images are scenes that usually depict significant depth, including paths leading into the hills with animals or people in the distance.
Sarch et al. () explored the representation of image depth in cortex and also found that similar areas of cortex respond strongly to images with objects very close to the camera. In addition, Luo et al. () reported that this general area is sensitive to relatively large objects.

## 5 Conclusion

We proposed a novel mechanism to identify shared decodable concept clusters in a large multi-participant fMRI dataset of natural image viewing (Allen et al., ). To achieve this, we applied a contrastive learning approach using multimodal embeddings from CLIP (Radford et al., ) and a novel adaptation of the DBSCAN clustering algorithm (Ester et al., ). Our method identifies previously reported functional areas that code for various visuo-semantic categories, as well as identifying novel shared categories. Furthermore, our method exposes interpretability by identifying the most / least associated images with CLIP-based cluster centroids. In the case of faces, we find a set of positive / negative images where the negative images depict scenarios where a face might be expected, but is not visible. This suggests that the brain responses for this SDC cluster are organized in a way that specifically codes for the presence of faces in expected visual depictions. Food is a concept that is highly confounded with other visual criteria such as shape and color. By providing the most negatively associated images for a SDC cluster, we enabled easy interpretation of semantic coding, e.g. in the case where the negated images to this SDC cluster are all grayscale images (see Section 4.2). We further identify SDC clusters specific to bodies in motion, in groups, as well as hands and legs. We identify SDC clusters related to repeated elements (numerosity), sensitivity to indoor / outdoor scenes, cardinal orientation of objects in images (horizontal / vertical) and gradations of shape information (soft vs hard) and visual perspective (close vs far). Our method showcases how various techniques from machine learning can be employed to better characterize visual representations of semantic selectivity in the human brain.

### 5.1 Broader Impacts

Methods that decode neural activity could have substantial impacts on both neuroscience and broader society. Identifying new category-selective brain regions could assist in diagnosing neurological and psychiatric conditions, surgical planning, development of brain-computer interfaces, and potentially helping individuals with locked-in syndrome or related disabilities to better communicate.

However, there are also significant ethical considerations with technologies that decode neural responses. If these techniques are used outside controlled research environments, there is a risk that they could be used for intrusive monitoring of mental states. As with all emerging technologies, responsible deployment is needed in order for society as a whole to maximize benefit while mitigating risks.

### 5.2 Limitations and Future Work

While our approach offers many advantages, it is not without drawbacks. Firstly, we are limited by the stimulus images that were chosen for the NSD experiment. Although the use of a contrastive loss function helps mitigate this, our model may still be biased toward over-represented categories in the stimulus set. Additionally, the use of CLIP may introduce its own biases, as some signal in the brain responses may not be fully captured by the CLIP embeddings. The use of fMRI also presents certain limitations. Despite its high spatial resolution, its limited temporal resolution restricts the ability to capture rapid neural activity that may play a crucial role in visual processing. Future work could address these limitations by utilizing larger and more diverse fMRI datasets, incorporating higher temporal imaging techniques such as electroencephalogram (EEG) or electrocorticography (ECoG), and exploring alternative stimulus representations beyond CLIP.

Additionally, our clustering approach might merge concepts within regions of CLIP space that have relatively uniform densities, potentially missing interesting SDCs. This issue could be addressed by adapting the heirarichcal version of DBSCAN, or by using different criteria to merge the cross-participant core points into clusters.

Furthermore, we acknowledge that our method is not a replacement for traditional hypothesis-driven experiments. It will be important for future work to test for alternative hypotheses for what might be driving responses in SDC clusters. Another interesting research direction is the application of this method to other sensory modalities beyond vision, such as auditory or somatosensory stimuli.

## 6 Supplementary Material

### 6.1 Decoding CLIP Representations

Figure: Figure 11: Decoding CLIP representations from the brain. We create CLIP representations by passing NSD training set stimuli to CLIP (frozen). The NSD team showed these same images to human participants during fMRI scans. We train a linear decoder with contrastive loss (Eq. 2) to predict CLIP embeddings from the fMRI responses to the corresponding images. Black arrows represent the flow of data through the procedure and dashed red lines represent gradient updates used to train the model.
Refer to caption: extracted/5892400/DemoFigure.png

The decoding model is trained for 5000 iterations (29 to 45 epochs depending on the participant’s training set size) with the Adam optimizer, a batch size of 128, and a fixed learning rate of $1e^{-4}$. Data augmentation is applied to help slow overfitting by adding random noise to training samples $\bm{x}_{i}\leftarrow\bm{x}_{i}+\bm{z}$ sampled from a normal distribution $\bm{z}\sim\mathcal{N}(0,\sigma^{2})$ where the noise standard deviation $\sigma$ is a hyper-parameter. We set $\tau=0.03$ and $\sigma=0.1$ in our implementation. Hyper-parameters were selected based on performance on the validation set. The exact number of voxels input to the model was $[17883,18358,13476,11899,17693,18692,9608,6772]$ for subjects 1 through 8. We compare our contrastive decoder to a baseline ridge regression model trained on the same data. We used grid search to select the best ridge regularization parameter $\lambda\in\{0.1,1,10,100,1000,10000,100000\}$ using the validation data. The optimal $\lambda$ was 10000 for participants 1, 2, 5, 6, and 1000 for participants 3, 4, 7, 8.

### 6.2 Compute Resources

The decoder models were trained on an NVIDIA GeForce RTX 2060. Training time was approximately 2 minutes for each model. The total training time for 50 models for each of the 8 NSD participants was about 13 hours. The DBSCAN clustering algorithm uses minimal CPU resources and executes within a few minutes.

### 6.3 Region of Interest identification

In order to localize shared voxel clusters in our visualizations, we overlaid region of interest boundaries onto the flatmap visualizations in Freesurfer’s fsaverage space. For functional ROIs, we retraced the ROIs given in the NSD dataset during the functional localization experiments (fLoc, Stigliani et al. ()).

### 6.4 Top-k Accuracy for CLIP Decoding

Figure: Figure 12: Top-k accuracy for CLIP decoding using the proposed contrastive decoder and a ridge regression baseline. The contrastive decoder implementation outperforms the ridge regression baseline across various values of $k$. Chance performance is given in green. Accuracy calculated on held-out data. Bars indicate the standard error (SE) across the 8 NSD participants. Additionally, we include a comparison to Liu et al. () who report the mean top-k accuracy across 4 subjects (1, 2, 5, 7) using brain responses to 982 test images (dashed red line). We average the results of our own contrastive decoder across these participants for a fair comparison (dashed blue line). Despite a slightly larger test set, our contrastive decoder has a higher accuracy for all reported values of k.
Refer to caption: extracted/5892400/top_knn_accuracy_v9.png

### 6.5 Voxel Normalization

Brain responses were per-voxel normalized to have zero mean and unit standard deviation within each scanning session. As a result, we allow brain response values to be positive or negative. In order to study both positive and negative aspects of stimuli-driven brain responses, this is a necessary preprocessing step as the beta weights in NSD are calculated with respect to a baseline gray background which is not zero-centered (Allen et al., ).

Figure: Figure 13: Cluster 3 ($\varepsilon=0.55$). Positive images display crowds, while negative images are of individuals. Voxel clusters are in bilateral EBA.
Refer to caption: extracted/5892400/bodies_repeated_epsilon0.55_cluster3.png

### 6.6 Places

When $\varepsilon=0.55$, cluster 9 (Figure 17) strongly overlaps with PPA and displays outdoor scenes with consistent vegetation for the positive images. The negative images show indoor scenes where human-made objects are prominent, with a distinct lack of vegetation.

At $\varepsilon=0.6$, cluster 4 (Figure 21) displays indoor scenes for the positive representative images. This cluster is mostly localized in OPA with some voxels in PPA. The positive images depict cluttered indoor scenes such as kitchens, work spaces, and living rooms. There is typically a flat surface such a desk, countertop, or table at the focal point of the image. The negative images depict outdoor scenes with animals.

### 6.7 Bodies (continued)

In Section 4.3 we identified various semantic features relating to the presence of bodies in images. We expand on a few further observations here. Cluster 3 (Figure 13) shows groups of three or more people in the positive images, with a strong focus on an individual person, animal, or object in the negative images. Cluster 6 (Figure 14) appears to be related to full-body leaping motions, with the negative images showing people sitting or standing still. In contrast, cluster 5 shows a mixture of images of people who are crouched or sitting, along with images of pet cats and dogs laying down.

### 6.8 Food

The positive images for cluster 3 (Figure 16) are mostly images of food. However, we observed themes of softness, round objects, and animals shared across the food and non-food images. Meanwhile, the negative images depicted many rigid and boxy objects such as trains, busses, and city buildings. This supported the idea that this cluster responds to a soft versus hard concept and is not entirely food-specific.

Cluster 5 (Figure 15) is shared across seven participants and is localized to frontal brain regions. Voxels are located in the orbital sulci, as well as the boundary between the triangular part of the inferior frontal gyrus and the inferior frontal sulcus. We noted that the positive images to contain images of food on plates, as well as a few non-food images (bear, skiing, playing baseball). We had trouble discerning a strong core concept in the negatively associated images, aside from a distinct lack of food.

Figure: Figure 15: Cluster 5 ($\varepsilon=0.65$). Food-related images with shared voxel clusters in anterior regions of the brain.
Refer to caption: extracted/5892400/food_frontal_epsilon0.65_cluster5.png

Figure: Figure 17: Cluster 9 ($\varepsilon=0.55$). Positive images display scenes with plants and foliage, while negative images show human-made objects. Voxel clusters are in bilateral PPA.
Refer to caption: extracted/5892400/outdoors_nature_epsilon0.55_cluster9.png

### 6.9 Soft vs Hard Cluster

We observe two clusters that relate to soft versus hard objects. When $\varepsilon=0.60$, cluster 8 shows images with a strong focus on clothing, bedding, and other textiles. The negative images show trains, and concrete-dominant architecture. The voxels in this cluster are mostly found in the right hemisphere bordering the area between FFA and EBA. As discussed in section 4.2, cluster 3, $\varepsilon=0.65$ displays similar negative representative images, while the positive images are mostly soft foods (Figure 16).

Figure: Figure 18: Cluster 8 ($\varepsilon=0.60$). Positive images contain textiles, bedding, and other soft things. Negative images show hard objects. Voxel clusters are primarily in the right hemisphere between EBA and FFA.
Refer to caption: extracted/5892400/soft_epsilon0.60_cluster8.png

### 6.10 Lighting-Related Cluster

When $\varepsilon=0.5$, most clusters are smaller or disconnected versions of clusters seen at higher $\varepsilon$ values. The exception is cluster 7 (Figure 19), which is unique to $\varepsilon=0.5$ and includes voxels in RSC and PPA. The positive images depict scenes with a high contrast in lighting. There is typically a dark environment with a bright light or a window that is partially illuminating the room. Conversely, the negative images depict close-up pictures of objects with uniform ambient lighting. This strongly suggests that this cluster of voxels responds to images that display a high contrast in lighting.

### 6.11 Words

Words and signs appear in NSD, and so we were interested to see if a word-related concept would emerge from our technique. When $\varepsilon=0.60$ we found cluster 11 to have images with objects with characters like signs and clocks. The negation of this concept is perhaps best characterized as having strong visual contrast over large areas of the image (e.g. a backlit traffic light or a white desk). This cluster is bilaterally localized to the borders of V4

## 7 Representative Word Clouds

Similar to selecting representative images, we can find sets of representative words for the SDC clusters. To accomplish this we first embed all 5 captions for each of the 73000 images in the NSD stimulus set to obtain $\bm{Y}_{\text{text}}\in\mathbb{R}^{365000\times 512}$. Unlike the representative images, these embeddings are not brain-decoded and are generated solely by the CLIP text encoder. To select the best representative captions we compute $D(\bm{\bar{w}},\bm{Y}_{\text{text}})$ and $D(-\bm{\bar{w}},\bm{Y}_{\text{text}})$ and select the captions corresponding to the 50 smallest distances in each set as the positive and negative captions respectively. We then utilize the wordcloud python package to create visualizations of the most frequently occuring words in the captions, which we present in the following chapter.

In Figure 20, we present word clouds containing the most frequent words in the representative captions for each image. In most cases the word clouds support our interpretation of each underlying cluster concept. For example the word cloud for the lighting cluster (Figure 19) containing the words lit, lamp, sun, illuminated, dark, dimly, brightly, all of which are indicative of lighting conditions. Similarly, the repeated elements cluster (Figure 9) is well-characterized by words like group, together, formation, squadron, herd, which emphasize the concept of repetition or grouping.

In some cases, while the word clouds provide partial support for our descriptions, the thematic patterns become more apparent when examining the representative images. For example the positive representative words for the hands cluster (Figure 6) contains action-oriented words like as cutting and slicing, and the legs cluster (Figure 5) features words such as riding and running. However it is difficult to identify the overall theme of hands or legs without the representative images.

The word clouds for the food clusters offer additional insights. The food and color cluster (Figure 4) is strongly indicative of favoring the color versus grayscale concept. In contrast, the anterior food cluster (Figure 15) has a strong association to food-related terms like food, eating, plate, avocado, and feeding. However, the food and softness cluster (Figure 16) lacks food-specific words despite an evident theme of food in the representative images. Although there are some softness-related words such as sheep, fur, slush, and snowy, there is not a strong theme of softness in the word cloud. These observations highlight the complexity of interpreting semantic representations in data-driven analysis, motivating the importance of targeted, hypothesis-driven experiments. For example, an fMRI experiment could be designed to test the neural responses to food stimuli compared to alternative categories like softness or color.

Figure: Figure 20: Word clouds for all SDC clusters. The positive and negative representative words are displayed side-by-side along with a subset of the representative images. The text to the left identifies the corresponding flatmap figure and our interpretation of the SDC.
Refer to caption: extracted/5892400/wordclouds_iclr_v2.png

Figure: Figure 21: Cluster 4 ($\varepsilon=0.60$). Positive images show indoor scenes with desks and tables. Negative images show outdoor scenes with animals. Voxel clusters are primarily in bilateral OPA and PPA.
Refer to caption: extracted/5892400/indoors_epsilon0.60_cluster4.png

## Peer Reviews

confidence: 4
contribution: 2
initial_rating: 5
presentation: 4
questions: The authors frame their study in the context of previous work on identifying functionally localized brain regions. However, it is unclear how their methodology would specifically highlight concepts confined to a few visual areas versus those spread across the brain. Instead, the approach appears to identify concepts that are easily decodable from brain signals across participants, regardless of whether decoding relies on signals from a small localized area or broader, distributed regions.
soundness: 2
strengths: - The discovered novel selectivities observed in the study are quite interesting and potentially good hypotheses for future brain imaging studies 
- The perspective that both active and suppressed neural responses are useful for identifying brain regions tied to distinct visual concepts is somewhat unique and not something I've seen in prior studies 
- This method circumvents the need to align subjects data, while still being able to extract shared representational structure across participants
summary: This study aims to identify brain areas that respond to specific visual concepts, aiming to expand our understanding of functional localization beyond known areas like the fusiform face area. Using a model based on CLIP embeddings, the authors map brain responses to clusters of shared visual concepts, called "Shared Decodable Concepts" (SDCs), across multiple participants. By applying a modified DBSCAN clustering algorithm, they reveal consistent brain regions linked to both previously known (e.g., faces, places) and novel visual concepts (e.g., specific body parts, numerosity, perspective). This method particularly allows data analysis within each participant’s unique brain structure, avoiding issues with traditional alignment methods.
weaknesses: - More methodological details are needed. More motivation behind why DBSCAN clustering used over other clustering algorithms? How do the authors select the epsilon which ultimately determines the number of clusters? 
- The authors should acknowledge that there is a lot of human bias involved in interpreting the concepts based on the activated images. For e.g. in 4.1, the authors interpret the negative representative images to indicate images that lack clearly visible faces. However, many other interpretations are also consistent with these sets of images (e.g. they all contain scenes). Similarly Cluster 2 in Fig. 5 could mean something other than bodies too (implied motion?). The subjectivity involved in interpreting the clusters makes me less enthusiastic about the study. Automating this interpretability analyses in future work could be interesting. 
- To better understand the strengths of the proposed methodology, it would also be helpful to compare it against natural candidates for understanding shared representational structure across participants, like Canonical Correlation analysis

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 3
questions: - One main question I am left with is: how many of these proposed clusters contain novel interpretable forms of feature tuning? It is difficult to know without a much more systematic attempt to cite relevant literature on visual category selectivity. 
- It remains unclear to me whether cross-subject analysis is required to derive these forms of insight, as discussed above. 
- Another key question relates to the issue of bias mentioned above. It would be straightforward to attempt to replicate some of these analyses using BOLD5000, THINGS, or other large datasets to address the possibility of bias that arises from the NSD stimuli. Another critical issue is whether relying on the CLIP feature space unavoidably biases the outcomes of analyses toward semantically interpretable outcomes. In other words, does the use of CLIP cause us to "overfit our understanding" of these brain areas to a feature space that was constrained to be language-aligned? I am not positive how I would go about addressing this issue, but at minimum I would appreciate if the authors attempted to discuss it. 
- Given that orientation tuning is a hallmark of V1 representations, why did V1 not show up in the corresponding cluster(s), only V2/V3?

Overall, given the extent of concerns listed above, I am unable to rate the paper higher than a 3 currently. In principle, I am willing to raise my review score if the authors are able to better situate this work in the context of other literature, and more clearly pinpoint the specific novel findings/predictions that differentiate it from other work that has used CLIP to try to understand NSD.
soundness: 3
strengths: - The idea of clustering different subjects' parameter/weight vectors as a means of studying what is common (or different) across individuals' brains is a promising direction that merits more attention and exploration in the field. It's a clever way to overcome the challenges of aligning different brains, while respecting the nuances of individuals' representational signatures. 
- Beyond the well-studied forms of category selectivity that pop out in the SDC analysis, some of the observations are new and interesting (e.g. a "softness" cluster). Identifying these signatures was only possible because of the sophisticated overall methodological approach applied to a rich dataset. These provide provocative directions for future research, to be sure they are actually replicable signatures across datasets rather than clusters that are biased by the specifics of NSD and/or the use of CLIP. 
- The authors make use of the richest possible data (NSD) available, and I appreciate the careful treatment of noise ceiling-related issues in the paper. It's also important that the authors ran a control comparing CLIP decoding to ridge regression. 
- It's useful that this work provides further empirical perspective on the hotly debated topic of whether food selectivity is separable from color representation in the ventral stream. 
- It's powerful to provide clear visual intuition of both the positive and negative examples that correspond to each cluster, as a way to bolster both the interpretability and the hypothesis space that each SDC provides. 
- I deeply appreciate that the authors discussed the possibility of bias in the conclusion, which might arise from the use of NSD stimuli, and which might arise from the use of CLIP.
summary: The authors propose an approach for detecting "Shared Decodable Concepts" in human visual fMRI data: clusters in CLIP space that are decodable from common sets of voxels across multiple participants. These clusters are identified using a modified implementation of the DBSCAN clustering algorithm, which identifies clusters in high-dim spaces. In particular, the algorithm operates over vectors of linear decoding weights that map between voxels and CLIP latents. Each weight vector is treated as a brain-decodable concept for a given voxel, such that cosine distance between the weight vectors corresponding to different voxels from different subjects can be used to index the similarity of the concepts encoded therein. After clustering, the authors interpret the SDCs by inspecting a set of "positive" representative images (and words) that are associated with the centroid of each one, and by further identifying groups of "negative" images (and words) that are represented in a maximally different manner. These analyses enable the authors to identify many well-studied aspects of visual cortical organization, such as areas that are tuned to face, body, scene, and word content, and to identify some less-well-studied features such as crowds and lighting. These observations may form the basis for future hypothesis-driven studies that seek to further probe the nuances of the representations of these visual concepts.
weaknesses: - In general the paper does a poor job citing other relevant literature, missing some critical papers on category selectivity in the visual system, as well as on recent efforts to use CLIP models for data-driven interpretability of visual feature tuning. Given the topic of the paper, the fact that it has only 20 citations total is insufficient to cover recent work in this area. For example, the authors should connect their observations of body-related selectivity to other works such as: https://www.nature.com/articles/s41467-020-16846-w, https://www.nature.com/articles/s42256-023-00753-y, and perhaps others.  Citing few works beyond the "original" papers identifying FFA/PPA etc makes it hard to understand how many of the findings are novel versus replicates of other studies. I was also surprised that the authors did not cite/discuss the BrainDIVE paper (https://arxiv.org/pdf/2306.03089) and any other works that use NSD and CLIP to propose new kinds of representational distinctions and signatures in the visual system. Furthermore, Stan Dehaene has relevant work on numerosity representations in the brain that should be cited/discussed. The discussion of food/color representations is missing a key citation from Pennock et al (https://www.cell.com/current-biology/pdf/S0960-9822(22)01893-0.pdf)... and so forth. I'd also recommend discussing existing methods of functional / representational alignment that do not required projecting individuals into a shared anatomical space, such as hyperalignment (Haxby) and shared response modeling (Chen). Why would simpler methods like these be insufficient for this problem of revealing shared dimensions and visualizing the images that project strongly/weakly onto them? I believe that there's extra "juice" to be gained from the CLIP-based approach, it would just help if this historical context were touched on somewhere in the paper. 

- There are many ways to achieve high distance from a given cluster centroid in high dimensional space. I understand the reason for looking at the most- and least-associated images with each SDC cluster, but it's not clear that being atypical in a clustering sense implies that those images would actually have a suppressive impact on the voxels' or regions' response profiles. The results here provide clear testable hypotheses, but the impact of the work is limited without further data collection to validate/falsify the positive-negative axes the authors propose. 
- The introduction should convey why combining data across participants is a necessary step to discover new forms of category selectivity - this is not obvious. Would a straightforward clustering algorithm applied to the voxel tuning within individual subjects reveal many of these same clusters? The authors do not present analyses that clearly show that cross-subject analysis is necessary to uncover particular forms of interpretable latent dimensions. 

- The clarity could have been improved by streamlining the extensive methods description that occurs before any main results are presented, leaving the nitty-gritty details for the supplement. Sections 2 and 3 place a high working memory burden on the reader - reducing this could help a lot with readability. 

- This might seem like a minor/pedantic point, but the authors' use of the word "semantic" to describe feature tuning in high-level visual cortex is a bit fraught - even if different regions have selectivities that seam meaningful to the human eye, that does not mean that the neurons themselves necessarily compute semantically interpretable features. (Furthermore, language fMRI researchers likely also have a very different conception of what "semantically tuned patches of cortex" means, compared to vision researchers.) I would recommend that the authors reconsider wordings such as "semantic concepts that drive activity" or "semantically-tuned patches" throughout the paper, in favor of language that describes forms of "visual feature tuning" that may be semantically interpretable, post-hoc, without implying that the underlying cells themselves "know" what semantics are.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: Recommendations to improve the paper:

1. At least cite some of the relevant prior literature as discussed above.
2. It would be useful if the authors could clarify whether the "suppressed" responses are simply lower than those to other stimuli, or whether they are actually lower than a minimal baseline (e.g. the response when no stimulus is presented). In this reviewer's opinion "suppression" should refer to a response below the no-stimulus baseline, and it is not clear that is the case here.
3. The claim that food-related voxel clusters " span bilateral FFA, V4, and PPA." doesn't make sense given the usual definition of FFA and PPA as face and place selective. Presumably what is mean here is that food-related voxels are found near the FFA and PPA.
4. The authors should show how well the brain data reproduces CLIP embeddings (and how this changes over training). That is, show how similar Y_hat is to Y_CLIP. 
5. How does using the CLIP dimension (w_i) improve on just using the neural response values (x_i) in this method? 
6. Perhaps comment on whether these results tell us more about the brain or about CLIP.
soundness: 4
strengths: Strengths of the paper are that it tackles an important problem, applies it to a beautiful data set, and reports interpretable results that either serve to establish the validity of the method (via replication of established findings) or report intriguing new results. Another strength is the emphasis on the lowest responses of functional clusters, something that is widely appreciated in the field but rarely if ever explicitly discussed. A third strength is the care that was taken to re-calculate the per-voxel noise ceiling estimates on training data only, a small but important point.
summary: This paper proposes a new method to discover semantically interpretable fine-grained functional organization of fMRI responses that are consistent across participants. It then applies this method to the NSD data form Allen et al, reporting both established functional responses (FFA etc and responses claimed to be novel (e.g. for particular body parts, numerosity, etc).

I recommend acceptance of the paper because the method proposed is useful, the writeup is clear, and the analyses appear solid.
weaknesses: The main weaknesses of the paper concern the failure to situate the paper in the context of relevant prior work. Some of the results are described as novel without reference to related findings published previously, For example, the claim of selectivity for different aspects of the body should acknowledge a large prior literature on this topic, e.g.: 
Orlov, T., Makin, T. R., & Zohary, E. (2010). Topographic representation of the human body in the occipitotemporal cortex. Neuron, 68(3), 586-600.
Weiner, K. S., & Grill-Spector, K. (2011). Not one extrastriate body area: using anatomical landmarks, hMT+, and visual field maps to parcellate limb-selective activations in human lateral occipitotemporal cortex. Neuroimage, 56(4), 2183-2199.
Bracci, S., Caramazza, A., & Peelen, M. V. (2015). Representational similarity of body parts in human occipitotemporal cortex. Journal of Neuroscience, 35(38), 12977-12985.
Bracci, S., Ietswaart, M., Peelen, M. V., & Cavina-Pratesi, C. (2010). Dissociable neural responses to hands and non-hand body parts in human left extrastriate visual cortex. Journal of neurophysiology, 103(6), 3389-3397.
big-small close/far this is the ref:
Konkle, T., & Oliva, A. (2012). A real-world size organization of object responses in occipitotemporal cortex. Neuron, 74(6), 1114-1124.
And concerning the proposed method, data-driven analyses of fMRI data especially using clustering have been reported for at least 15 years, and this should be at least acknowledged, e.g.:
Golland, Y., Golland, P., Bentin, S., & Malach, R. (2008). Data-driven clustering reveals a fundamental subdivision of the human cortex into two global systems. Neuropsychologia, 46(2), 540-553.
Lashkari, D., Vul, E., Kanwisher, N., & Golland, P. (2010). Discovering structure in the space of fMRI selectivity profiles. Neuroimage, 50(3), 1085-1098.
Yeo, B. T., Krienen, F. M., Sepulcre, J., Sabuncu, M. R., Lashkari, D., Hollinshead, M., ... & Buckner, R. L. (2011). The organization of the human cerebral cortex estimated by intrinsic functional connectivity. Journal of neurophysiology.
and contrasted with alternative data-driven methods using ICA, NMF, etc, e.g.:
Norman-Haignere, S., Kanwisher, N. G., & McDermott, J. H. (2015). Distinct cortical pathways for music and speech revealed by hypothesis-free voxel decomposition. Neuron, 88(6), 1281-1296.
Khosla, M., Murty, N. A. R., & Kanwisher, N. (2022). A highly selective response to food in human visual cortex revealed by hypothesis-free voxel decomposition. Current Biology, 32(19), 4159-4171.

---

confidence: 4
contribution: 3
initial_rating: 5
presentation: 3
questions: - Can you explain this sentence: "We emphasize that there is no constraint that voxels within a shared visual semantic cluster be spatially adjacent in the brain". 

- How does this paper go above and beyond the findings of other recent works exploring functional selectivity using the Natural Scenes Dataset?


- Can you provide some intuition for why the contrastive fitting of voxel -> clip latents outperforms ridge regression?
soundness: 3
strengths: - Interesting method
- The use of a contrastive loss in the fitting of clip latents from voxel activities is particularly interesting
- The modification of DBSCAN is elegant, and provides an intriguing way of aggregating information across subjects
- Good figures
summary: This paper presents a novel technique for discovering and interpreting functionally specialized clusters in the brain. Using the Natural Scenes Dataset, the authors train linear decoders to map single voxel activities onto CLIP embedding vectors for the respective images seen in the neuroimaging experiment. Then, the authors leverage the similarity of voxel->CLIP prediction weights to perform clustering. Specifically, the authors develop a variant of the DBSCAN algorithm that leverages across-subject consistency in order to find the most consistent and semantically meaningful components. The authors present the results for several components, showing the spatial topography as well as the most strongly and weakly activating images. The authors interpret the components in light of both their spatial topography, and the content that appears present in the strong and weak activating images.
weaknesses: - The early vision components (horizontal and straight lines) activate parts of V1-V3 that represent the periphery (if the authors are unaware: on the flat map, the center of the strips of V1-V3 corresponds to foveal bias, and moving outward corresponds to increasing peripheral eccentricity). This isn't discussed but seems important. 

- The results are very qualitative. The paper could be strengthened by making some objective quantifications of the content of different components, rather than just "reading the tea leaves". 

- This paper joins with a serious of related papers exploring visual components (e.g. Khosla et. al 2022; BrainDIVE, Luo et. al 2023 NeurIPS; BrainSCUBA, Luo et. al 2024 ICLR), to name a few. I'm not sure if there is any single result here that is particularly novel. This makes the paper less compelling. Additionally, the method is not compared to other methods, so it is unclear whether it holds any advantages. 

- The paper is very light on references. In my opinion, having 20 references for a paper on such a popular topic is a bit of a disservice to the field. 

- This sentence is confusing: "We emphasize that there is no constraint that voxels within a shared visual semantic cluster be spatially adjacent in the brain, yet during visualization we consistently find contiguous patches both within and across participants". Please explain how this can be. The algorithm appears spatial, in that core points are defined as having at least minNeighbors points within their epsilon (spatial) neighborhood. Additionally, the epsilon expansion appears to be highly spatial in nature. If it is truly the case that there is no spatial constraint, it would be a mistake to simply call the algorithm a DBSCAN variant -- it would be better to say that it is inspired by DBSCAN, then give it a new name to avoid confusion. Actually, this is probably a good idea regardless.

Meta-review: This paper introduces a contrastive model that maps brain responses during naturalistic image viewing to CLIP embeddings, revealing Shared Decodable Concepts (SDCs)—clusters of visual-semantic features linked to common brain regions. Using a novel adaptation of DBSCAN, the study uncovers known and previously unreported visuo-semantic sensitivities, such as distinct food-related clusters and areas tuned for hands/legs or numerosity. This methodology advances the characterization of visuo-semantic representations in the brain by integrating multimodal neural networks with innovative clustering techniques.

All the reviewers agree about the interest and merits of the paper. Substantial improvments have also been approved during the rebuttal period. In this light, I am recommending the paper for acceptance at the ICLR 2025 conference

Decision: Accept (Poster)