Finding Shared Decodable Concepts and their Negations in the Brain
Authors: Cory Daniel Efird
Alex Murphy
Joel Zylberberg
Alona Fyshe
Venue: ICLR 2025 Poster (2025)
Keywords: fMRI
decoding
computer vision
neuroscience
Primary area: applications to neuroscience & cognitive science

Abstract: Prior work has offered evidence for functional localization in the brain; different anatomical regions preferentially activate for certain types of visual input. For example, the fusiform face area preferentially activates for visual stimuli that include a face. However, the spectrum of visual semantics is extensive, and only a few semantically-tuned patches of cortex have so far been identified in the human brain. Using a multimodal (natural language and image) neural network architecture (CLIP, \cite{CLIP}, we train a highly accurate contrastive model that maps brain responses during naturalistic image viewing to CLIP embeddings. We then use a novel adaptation of the DBSCAN clustering algorithm to cluster the parameters of these participant-specific contrastive models. This reveals what we call Shared Decodable Concepts (SDCs): clusters in CLIP space that are decodable from common sets of voxels across multiple participants.

Examining the images most and least associated with each SDC cluster gives us additional insight into the semantic properties of each SDC. We note SDCs for previously reported visual features (e.g. orientation tuning in early visual cortex) as well as visual semantic concepts such as faces, places and bodies. In cases where our method finds multiple clusters for a visuo-semantic concept, the least associated images allow us to dissociate between confounding factors. For example, we discovered two clusters of food images, one driven by color, the other by shape. We also uncover previously unreported areas with visuo-semantic sensitivity such as regions of extrastriate body area (EBA) tuned for legs/hands and sensitivity to numerosity in right intraparietal sulcus, sensitivity associated with visual perspective (close/far) and more. Thus, our contrastive-learning methodology better characterizes new and existing visuo-semantic representations in the brain by leveraging multimodal neural network representations and a novel adaptation of clustering algorithms.

Published as a conference paper at ICLR 2025
FINDING SHARED DECODABLE CONCEPTS AND THEIR NEGATIONS IN THE BRAIN
Cory Efird
Alex Murphy
Computing Science University of Alberta
Computing Science
University of Alberta
Alberta Machine Intelligence Institute (Amii) e wedeualbeetaa...mm
Alberta Machine Intelligence Institute (Amii)
mourphyjqualbert..o.
Joel Zylberberg
Alona Fyshe
Physics and Astronomy
Computing Science and Psychology University of Alberta Alberta Machine Intelligence Institute (Amii)
York University
joel zy@yorku.ca
a alana @ua berta. ca
ABSTRACT
Prior work has offered evidence for functional localization in the brain; different anatomical regions preferentially activate for certain types of visual input. For ex- ample, the fusiform face area preferentially activates for visual stimuli that include a face. However, the spectrum of visual semantics is extensive, and only few semantically tuned patches of cortex have so far been identified in the human brain. Using a multimodal (natural language and image) neural network architecture (CLIP, Radford al. (202T)), train highly accurate contrastive model that maps brain responses during naturalistic image viewing to CLIP embeddings. We then novel adaptation of the DBSCAN clustering algorithm to cluster the parameters of these participant-specifie contrastive models. This reveals what we call Shared Decodable Concepts (SDCs): clusters CLIP space that are decodable from common sets of voxels across multiple participants.
Examining the images most and least associated with each SDC cluster gives us additional insight into the semantic properties of each SDC. We note SDCs for previously reported visual features (e.g. orientation tuning in early visual cortex) as well as more complex concepts such as faces, places and bodies. We also uncover previously unreported areas with visuo-semantic sensitivity such as regions of extrastriate body area (EBA) tuned for legs/hands and sensitivity to numerosity in right intraparietal sulcus, sensitivity associated with visual perspective (close/far) and more. Thus, contrastive leamning methodology better characterizes new and existing visuo-semantic representations in the brain by leveraging multimodal neural network representations and novel adaptation of clustering algorithms.
INTRODUCTION
The study of the visual system has revealed a significant amount of knowledge about how the brain processes and organizes visual information. The early visual system has been well characterized for its ability to extract low level visual features (c.g. lines of different orientations). However, the picture becomes more murky as one moves beyond the early visual system. There is evidence for functional localization wherein large patches of cortex activate for specific visuo-semantic properties: areas of the brain preferentially activate for visual stimuli containing faces (Kanwisher et al., 1997). places (Epstein and Kanwisher 1998), bodies (Orlov et al. 2010 Weiner and Grill-Spector 2011
Bracci et al.] 2015 2010). There have been multiple recent attempts to identify new functionally localized regions of cortex with some converging evidence; for example, food near PPA has been identified (Khosla et al.) (022). So, while there has been progress towards parcelating the functional regions of the brain, only few such broad visuo-semantic categories have been identified

---

Published as a conference paper at ICLR 2025
and they typically cover fairly large areas of cortex. This leaves open the possibility that we may be able to detect new visuo-semantic categories in the brain or refine existing categories, and that these categories or cub-categories may become more apparent with data-driven approach that can combine data across multiple participants.
But combining brain responses across participants is not simple. Prior research typically focused on mapping to a shared latent space (e.g. during stimulus reconstruction) for which spatial localization in the brain is not required. In cases where spatial information is required, analyses are typically applied to data which has been warped to a common template brain, often losing the connection to the unique structure of each participant's 's cortical layout. Prior work using clustering specifically of fMRI data is well represented in studies such as Golland et al. (2008): Lashkari et al. (2010); Thomas Yeo al. (201T), though alternative methods based on independent component analysis (ICA) have also helped to advance understanding of FMRI selectivity
et al. 2015).
We present a novel method that finds semantically meaningful shared voxel clusters while still considering the voxels in each participant's 's native brain space, allowing to sidestep the voxel alignment problem. Derived voxel clusters can be mapped to shared space for visualization after identification of shared semantic clusters.
Our data-driven analysis of fMRI recordings during viewwng allows not only for identification of the images that most drive neural activity in given area, but also of images which are most associated with below-average BOLD response. This provides stronger evidence for the possible function of cortical areas, in some cases refining the hypotheses one might draw from a set of images. Brain areas showing below average in response to a stimuli may be consistent with recently reported "offsembles" of neurons (Pérez-Ortega et al., (2024) which show inhibited activity in response e.g. visual stimuli, implying that, when interpreting neural representations, below average activity may be as important as above average.
We identify shared decodable concepts (SDCs) that drive activity in patches of cortex across multiple participants. We find that these SDCs are semantically interpretable and that their localization is fairly consistent across participants. We identify SDCs with our novel variant of DBSCAN that combines data from multiple fMRI participants, specifically searching for SDCs present in multiple subjects.
DECODING CLIP-SPACE FROM BRAIN IMAGES
To identify shared decodable concepts (SDCs) in the brain, we derived mapping from anatomical brain space to a representational space in which we can explore semantic sensitivity to visual input. In this section we describe the components of this mapping: a dataset of fMRI recordings (NSD), a multimodal image-text embedding model (CLIP), and our method to map from per-ine ver-image brain responses their associated multimodal embeddings. We consider two decoding models in this section, and verify that our proposed contrastive decoder outperforms a baseline ridge regression model.
2.1 FMRI DATA
The natural scenes dataset (NSD) is massive fMRI dataset acquired study the underpinnings of natural human vision (Allen et al. 2022). Eight participants were presented with 30,000 images (10,000 unique images over 3 repetitions) from the Common Objects in Context (COCO) naturalistic image dataset (Lin et al. 2014). A set of 1,000 shared images were shown to all participants, while the other 9,000 images were unique each participant. Single-trial beta weights were derived from the fMRI time series using the GLMSingle toolbox (Prince et al. 2022). This method fits numerous haemodynamic response functions (HRFs) to each voxel, as well as an optimized denoising technique and voxelwise fractional ridge regression, specifically for single-trial fMRI acquisition paradigms. Some participants did not complete all fMRI recording sessions and three sessions were held out by the NSD team for the Algonauts challenge. Further details can be found in Allen et al. (2022)
2.2 REPRESENTATIONAL SPACE FOR VISUAL STIMULI
To generate representations for each stimulus image, we use CLIP (Radford et al. 2021), a model trained over 400 million text-image pairs with contrastive language-image pretraining objective.
2

---

Published as a conference paper at ICLR 2025
CLIP consists of text-encoder and image-encoder that jointly learns shared low low-dimensional space trained to maximize the cosine similarity of corresponding text and image embeddings. We use the 32-bit Transformer model (ViT-B/32) implementation of CLIP to create a 512-dimensional representation for each stimulus image used in the NSD experiment. We train a decoder to map from fMRI responses during image viewing the associated CLIP vector for that same image. Notably, because CLIP is a joint image-language model, these CLIP vectors also correspond to text captions in the pretraining stage, which can be used describe the images presented to the participants.
2.3 DATA PREPARATION
Data Split We split the per-image brain responses X and CLIP embeddings Y into training (Xrain, Yrain). validation (Xval, Yvai), and test (eeest, Yest) folds for each of the 8 NSD participants. For each participant, the validation and test folds were chosen have exactly 1,000 images with three presentations. Of the shared 1,000 images that all participants saw, 413 images were shown three times to every participant across the sessions released by NSD. These 413 images appear in each participant's testing fold.
Voxel Selection The NSD IMRI data comes with voxelwise noise ceiling (NC) estimates that can be used identify reliable voxels. However, the NC is calculated using the fill dataset. Therefore, these NC estimates should not be used to extract subset of voxels for decoding analyses because they are calculated using images in the test set, which a form of double-dipping (Kriegeskorte et 2009). We therefore re-calculated the per-voxel noise ceiling estimates specifically our designated training data only. We selected voxels with noise ceiling estimates above 8% variance explainable to use inputs to the decoding model. This resulted in 6k-19k voxel subsets per participant (see supplementary section A.2 for exact per-participant voxel dimensions input into the decoder) which were normalized according to section|.. In our visualizations, regions of the flattened brain surface in black represent voxels that have passed this voxel selection threshold.
2.4 DECODING METHODOLOGY
Decoding Model The decoding model g = Ý is a linear model trained to map brain Fn.] responses X = (/21,...,,.,,,, x; E IR" to brain embeddings Y [y yn), y: E 51522 Here n the number of training instances, and v is the participant-wise number of voxels that pass the noise ceiling threshold. We optimize the brain decoder using the InfoNCE definition of contrastive loss
(van den Oord et al. 2018), which is defined below in Equations
and
exp(a; bu/T) expla; b,/T)
Contrast B)
M Σ log
M Sj=1
LISONNCE A. B) CContrast A, B) + Contrast( B, B,
In this definition [ai,..., an) and (b..... b,] are embeddings for two modalities representing the same data points, the operator represents cosine similarity, and 7 is temperature hyper-parameter The loss is minimized when the distance between matching embeddings is small, and the distance between mismatched embeddings is large. In the original CLIP setting, A and B represent embeddings for images and corresponding text captions. our implementation, we apply the contrastive loss to image embeddings computed by a pretrained frozen CLIP model and CLIP embeddings predicted from IMRI, i.c. optimize mino L(Ý, YCUP) L(g(X; 0), Y). An illustration the decoding procedure is given Figure (additional details on the decoding procedure are given in Section|...
Evaluation We evaluate our models using top-k accuracy. Figure 12 shows the results of this evaluation. The contrastive decoder outperforms ridge regression across all values of k. Recall that
'Some participants in the NSD did not complete all scanning sessions and only viewed certain images once twice. These images are assigned to the training set, which varies in size across participants.
3

---

Published as a conference paper at ICLR 2025
our end goal is to identify shared decodable concepts (SDC) in the brain. Our methodology for this task relies on the predicted CLIP vectors, and so our subsequent analyses require an accurate decoding model.
3 FINDING SHARED BRAIN-DECODABLE CONCEPTS AND THEIR REPRESENTATIVE IMAGES
In this section, describe our methods for finding and interpreting areas of the brain that respond to semantically similar images, and that are consistent across participants. We approach this problem by analyzing the weight matrices (*)) from the optimized linear decoders described in section 3.4 g g( (x: (x) = y for all subjects l E (1...88 .8).. First, we observe that the decoder's linear transformation (k) = - w (k) xx.h (k) is simply summation of parameter vectors (k) w!! E 512 that are scaled by brain response values x; E R. This means that when x; is above or below baseline activation, the CLIP dimension represented by un!
(8) is increased or decreased in the brain-decoded embedding. This motivates the key to our analysis: we view the parameter vector
w, as CLIP vector that represents a decodaeec concept for voxel i. This allows to use cosine distance d(w, w between the weight vectors for voxels i, from participants k, r as measure of similarity between the decodable concepts of brain voxels across participants. Our objective is to apply a clustering algorithm using this metric in order to find areas in the brains of multiple participants that have high conceptual similarity. We refer to these underlying concepts as shared decodable concepts (SDCs). We interpret the SDCs by retrieving set of representative images associated with the brain-decoded CLIP vectors that are elosest to the centroid of each SDC cluster. A schematic of the algorithm we apply is given in Figure 10
(8)
3.1 CROSS PARTICIPANT CLUSTERING
Much of the prior work reporting semantically interpretable information in NSD focuses on modeling individual participants (e.g. Luo et al.) (b). This is because current methods that combine (MRI across participants without anatomical alignment, so-called functional alignment methods, require that participants see the same stimuli in the same order (e.g. Shared Response Model (Chen) al. 2015), Hyperalignment (Haxby al. 2020). However, in NSD image order is randomized and most images are seen only by a single participant, precluding the application of common functional alignment methods and other methods that require the same stimuli in each sample space (c.g.. Canonical Correlation Analysis, CCA). Our approach leverages participant specific models that cast each voxel into CLIP's feature space, allowing us to use data that (a) aren't anatomically aligned, and (b) don't 't share exactly the same stimuli sets.
To discover shared brain-decodable concepts we apply a novel clustering method to the vorev model parameter vectors (k)) across all participants. We base our clustering method on the Density- Based Spatial Clustering of Applications with Noise (DBSCAN) (Ester et al., 1996) algorithm, which is parameterized by a neighborhood size & and a point threshold minPts. The algorithm can be summarized using the following steps: (1) Points that have at least mi tPts points in their e-neighborhood are marked as core points; (2) construct a graph G where the vertices are core points, and there are edges between core points that have a distance less than E; (3) clusters are formed by finding the connected components of G and (4) the remaining non-core points are added to clusters they are within the E-neighborhood of a core point, otherwise, they are marked as outliers not belonging any cluster.
In our modified cross pooss-pariii DBSCAN, we redefine the core point threshold minPts as mi minNeighbors. Then, in step (1) point w
(k) is point if there are points from at least minNeighbors other participants within its e neighborhood. With this modification, a core point now identifies voxels that represent brain-decodable concept that is shared with at least minNe iigborg participants. Since there are participants in the study, the maximum value of minNe iilhlrgg is 7. Steps (2) to (4) proceed as above.
Our second modification is to apply a within authin-partcciannt expansion of clusters. We define a new hyperparameter Eexpansion and apply final expansion step: (5) All points inside the Eexpansion neigh-
4

---

Published as a conference paper at ICLR 2025
borhood of a point in a cluster become members of that cluster with the constraint that they must be from the same participant. The addition of this step was motivated by our early experiments, where noticed that there were sometimes only one two voxels belonging certain participants for each cluster. This allows the cluster boundaries to grow slightly larger for each participant, and we found that this did not significantly change the cluster centroid semantic interpretations. We set the expansion neighborhood size Ecxpansion min(s + 0.05, 0.65) so that the cluster boundaries can grow slightly larger than the baseline neighborhood size e, but not exceeding 0.65 we noticed neighborhoods become over-connected as approaches 0.7. Figure 1 outlines the application of our modified DBSCAN algorithm to modeling fMRI data.
We present results using a fixed value for mi minNeighhor.. = 3. This means that each cluster must include least out of participants in the study. This allows for the discovery of SDCs that may show up less reliably in some participants. Inspired by the hierarchical version of DBSCAN (HDBSCAN) Campello al. (213), we ran our modified DBSCAN algorithm over a grid of values, ranging from 0.8 to 0.4 in steps of 0.001. The resulting clusters are then organized into a hierarchical structure as defined by HDBSCAN, The first cluster considered contains all points at the highest value of e. As decreases, clusters may split into smaller subclusters. When cluster splits, classify its child(ren) clusters as follows: If two or more child clusters contain at least minClusters ze core points, they are added the hierarchy as distinet subclusters. If only one child cluster meets this threshold, it is treated as a continuation of the parent cluster, while the remaining smaller clusters (with fewer than minClust eettiiiii ze core points) are considered outliers. We show selection of clusters from the hierarchy in Section 4 and the appendix, while the full hierarchy of clusters can be explored in our interactive viewer tool (Section A.
To reduce the impact of random initialization and combat the noise inherent in IMRI, we re-trained the decoder model 50 times for every participant and compute the average of the resulting parameter vectors. In other words, each averaged parameter vector w, used in our analysis represents the average of 50 concepts that a voxel could represent depending on the random initialization of the decoder. We found that this significantly improved the quality and consistency of the clusters revealed by our modified DBSCAN algorithm.
b
binMeighbors 2
Outler points
sub11 SUbU02
SubIC3
Figure 1: Illustration of our DBSCAN variant applied to multi- cunti-partiiciantt fMRI data. There 3 participants in this example and mi nN eeiehhbrss with three core points. The outer ring around each point shows its g-neighborhood and whether it is a core, border, or outlier point. Black arrows emphasize points that are neighbors. The points with neighboring points from at least 2 other distinct participants are marked as core points. Non-core points that neighbor core points are added to the cluster as border points. The remaining points marked outliers. (b): A zoomed-oud -ut sketch of set of points that form 3 clusters. Since mi minNieghbors = 2, a high-density region will form clusters if and only if contains points from at least 3 participants.
(a): zoomed-in view of a cluster
3.2 SELECTING REPRESENTATIVE IMAGES AND WORDS
To interpret the semantic meaning of an SDC cluster, we first computed the cluster centroid by taking
the mean of all parameter vectors within the cluster v / Excnerr w (k) where / is set of tuples identifying the voxel and subject indices for SDC cluster. When the cluster voxels above
5

---

Published as a conference paper at ICLR 2025
below baseline activation, the CLIP dimension represented by w is increased or decreased in the brain-decoded embedding, respectively. We note that DBSCAN has the ability to create non- globular clusters for which the centroid might not be good representative. In experience the centroid is within & of a core point within the cluster, implying that it is likely fairly representative of the cluster.
In order find the representative images for SDC cluster, embeddings to the cluster centroid i. Instead of using the original CLIP embeddings for each image. instead used the subject-specifie brain decode embeddings (-(k) Test This allows to focus specifically on the information that can be decoded from fMRI, which could be a subset of the information represented by CLIP space. For images that viewed more than once average the embeddings within and across participants. These averaged prainddecoded embeddings from held-out test data are denoted by Y.AVG To select nearby images we define D(w, Y) = {d(w., y)ly E Y which is the set of cosine distances between CLIP vector ur set of CLIP embeddings Y. We take the cosine distances between the SDC cluster centroid D(w, Yest and the negated centroid D(-w. A and retrieve the images corresponding to the smallest values these sets as the positive and negative representative images respectively.
consider the nearby image-
Similar to selecting representative images, we can find sets of representative words for the SDC clusters. To accomplish this we first embed all 5 captions for each of the 73000 images in the NSD stimulus set to obtain Yeu E IR x165000x512 Unlike the representative images, these embeddings are not brain-decoded and generated solely by the CLIP text encoder. To select the best representative captions we compute D(w, Yest) and D(-w, Yieu) and select the captions corresponding to the 50 smallest distances in each set the positive and negative captions respectively. We then utilize the wordeloud python package to create visualizations of the most frequently occuring words in the captions, which we present in appendices (Figure 20
4 RESULTS
The clustering method described in chapter 3 not only identifies previously discovered functional areas but additionally identifies two new ones, both which we explore in this section. We emphasize that there is no constraint that voxels within an SDC cluster be spatially adjacent in the brain, yet during visualization we consistently find contiguous patches both within and across participants. We can localize these patches of voxels to specific regions of interest (ROIs) both functionally and anatomically (See Section A.4 for further details).
4.1 FACES
One of the first reported functionally localized areas for higher-order vision was the fusiform face area (FFA) (Kanwisher et al. 1997). Our method identifies a face-related concept (Figure cluster 88) that is localized to FFA and includes voxels from all 8 participants. This cluster also has some voxels in the extrastriate body area (EBA), likely because many face images include part or all of a person's body. We note that the positive representative images are exclusively human and include range of animal faces. The images often depict people cating and handling food, or holding other objects such as toothbrushes or cell phones.
Interestingly, the negative representative images often display bodies, but there is a striking lack of clearly visible faces. There are several examples where people are visible, but their faces are obscured or they are facing away from the camera. This suggests that there may be dip in FFA activity for images where person's 's face might be expected (and thus FFA is primed activate) but not clearly visible.
4.2 FOOD AND COLOR
Previous work has specifically noted the correlation of very colorful images with food-related images Khosla et al. (2022); Pennock et al. (2023), and attempted to define food areas in the absence of color (Jin et al. (223)). Our method also identifies a large possibly food-related cluster that spans FFA, PPA. and V4 (cluster 1 114, Figure B). At first inspection most of the positive representative images are food-related. However, we also observed many vibrant and colorful positive images that contained food. Strikingly, the negative representative images entirely gray-scale, providing

---

Published as a conference paper at ICLR 2025
Positive
Negalive
Positive
Negative
Figure Cluster 88 (E 0.593). Positive images are strongly associated with faces, while the nega- tive images represent depictions of people whose faces are not visible. Voxel clusters are primarily found in bilateral FFA and EBA.
Figure 3: Cluster 114 (5 0.555). Positive im-
ages are associated with food and color. Negative images are entirely grayscale. Voxel clusters span an area between bilateral FFA, V4, and PPA.
Positive
Negative
Negative
Figure 4: Cluster 106 (5 = 0.569). Positive im- ages are strongly associated with presence of legs, while negative images are typically people at ta- bles whose legs are obscured. Voxel clusters are primarily in bilateral EBA.
Figure 5: Cluster 89 (s 0.593). Positive images are strongly associated with hands and hand mo- tion, while negative images are associated with animals (no hands). Voxel clusters are primarily in bilateral EBA and FFA.
very strong evidence that this cluster may be related to color. Thus, we speculate that this cluster not specifically related the identification of food, but might rather correspond to any colorful image. We discuss two other food-related clusters in supplementary section A.9
4.3
BODIES
Our method identifies five notable body-related areas in and around EBA (Downing et al.) 2001 The positive representative images for cluster 106 (Figure 4 show people and animals outside with an emphasis on legs and active movement. The negative images typically depict people indoors sitting with their legs obscured by tables. Cluster 89 (Figure 5 has similar emphasis on hands instead of legs. People are displayed in variety of contexts with their hands clearly visible, while the negative images exclusively non-primate animals without hands. This cluster also shows indoor/outdoor contrast in the representative image groups and so there is some PPA activation. In Section A.8 we outline further body- rolytrel shared voxel clusters associated with motion and distinction between single individuals and groups of people in a sub-region of EBA.
4.4
HORIZONTAL/VERTICE CLUSTER
Very early work the visual system discovered the tuning of the early visual system for lines of particular orientation. Two clusters emerge that reflect this tuning. Cluster 60 (Figure has a strong horizontal component with strong horizon lines or large objects spanning the middle of the visual
7

---

Published as a conference paper at ICLR 2025
Positive
Negalive
Negative
Figure Cluster 60 0.668). Positive images are associated with horizontal mid-line, while negative images display a vertical mid-line. Voxel clusters are primarily in right hemisphere V2.
Figure 7: Cluster 65 (s 0.657). Positive images are associated with vertical mid-line, while neg- ative images display a horizontal mid-line, Voxel clusters are primarily in bilateral V3.
Positive
Negative
Positive
Negative
Figure 8: Cluster 111 (s 0.563). Positive im- ages are associated with repeated objects, people, or other animals, while negative images are single objects or individuals. Voxel clusters are primar- ily located in OPA and IPS.
Figure 9: Cluster 67 (E 0.654). Positive im- ages display close-ups of large animals or objects. Negative image show objects distance. Voxel clusters are located between EBA and V4.
field creating horizon-like line. Notably, the negative images are images with strong vertical component. The inverse is true for cluster 65 (Figure 7 which shows strong verticality in the positive images, and horizontal in the negative. Interestingly some of the negative images from cluster 60 appear as positive images in cluster 65, and vice versa. The localization of both clusters is primarily in the periphery regions of V2/V3. We hypothesize that the lack of foveal responses is related to the fact that horizontal and vertical paths, which define these orientation responses, likely cross in the foveal region. Therefore, peripheral responses expected to be much more informative in distinguishing the identified horizontal/vertical components of these images.
4.5 REPEATED ELEMENTS (NUMEROSITY) CLUSTER
We observed that cluster 111 (Figure 8) has positively associated images typically with repeated elements of related objects, people, or other animals in similar orientations, while negative images typically contained singular instances of objects or individuals. We suggest that this SDC could be related to quantity processing and numerosity. We observed voxel clusters primarily in right OPA and IPS regions. As with other clusters, naturalistic images often implicates functional areas associated with place (OPA). The IPS has been widely reported in prior work to be associated with quantity processing, e.g. grammatical number processing (Carreiras et al.] 2010). mathematical processing deficits (Ganor-Sters et al.] 2020 and numerical processing (Dehaene et al., 2003 Dehaene 2007] Nieder and Dehaene 2009: Koch al.) (202)). Our observations in the visual domain provide converging evidence that the right IPS is associated with quantity processing across multiple modalities.
8

---

Published as a conference paper at ICLR 2025
4.6 CLOSE VS FAR CLUSTER
Cluster 67 (Figure 2 seems to represent big things (airplanes, busses) and close up pictures of larger animals (zebras, elephants). The negatively associated images are scenes that usually depict significant depth, including paths leading into the hills with animals or people in the distance. [onkle and Oliva (0012) and |Sarch et al. (2023) explored the representation of image depth in cortex and also found that similar areas of cortex respond strongly to images with objects very close to the camera. In addition, [Luo
[Luo et al. (2023b) reported that this general area is sensitive to relatively large objects.
CONCLUSION
We proposed a novel mechanism to identify shared decodable concept clusters in large multi- participant fMRI dataset of natural image viewing (Allen al.. 2022). To achieve this, we applied contrastive learning approach using multimodal embeddings from CLIP (Radford et al. 2021) and a novel adaptation of the DBSCAN clustering algorithm (Ester et al. 1996). Our method identifies previously reported functional areas that code for various visuo-semantic categories, as well as identifying novel shared categories. Furthermore, our method exposes interpretability by identifying the most / least associated images with CLIP-based cluster centroids. In the case of faces, we find a set of positive / negative images where the negative images depict scenarios where a face might be expected, but is not visible. This suggests that the brain responses for this SDC cluster are organized in a way that specifically codes for the presence of faces in expected visual depictions. Food is a concept that is highly confounded with other visual criteria such as shape and color. By providing the most negatively associated images for a SDC cluster, we enabled easy interpretation of semantic coding, c.g. in the case where the negated images to this SDC cluster are all grayscale images (see Section 4.2) We further identify SDC clusters specific to bodics in motion, in groups, as well as hands and legs. We identify SDC clusters related to repeated elements (numerosity), sensitivity indoor / outdoor scenes, cardinal orientation of objects in images (horizontal vertical) and gradations of shape information (soft hard) and visual perspective (close far). Our method showcases how various techniques from machine learning can be employed to better characterize visual representations of semantic selectivity in the human brain.
5.1 BROADER IMPACTS
Methods that decode neural activity could have substantial impacts both neuroscience and broader society. Identifying new category-selective brain regions could assist in diagnosing neurological and psychiatric conditions, surgical planning, development of brain-computer interfaces, and potentially helping individuals with locked-in syndrome or related disabilities to better communicate.
However, there are also significant ethical considerations with technologies that decode neural responses. If these techniques are used outside controlled research environments, there is risk that they could be used for intrusive monitoring of mental states. As with all emerging technologies, responsible deployment is needed in order for society as a whole to maximize benefit while mitigating risks.
5.2 LIMITATIONS AND FUTURE WORK
While our approach offers many advantages, it is not without drawbacks. Firstly, we are limited by the stimulus images that were chosen for the NSD experiment. Although the use of a contrastive loss function helps mitigate this, our model may still be biased toward over-represented categories in the stimulus set. Additionally, the use of CLIP may introduce its own biases, as some signal in the brain responses may not be fully captured by the CLIP embeddings. The use of fMRI also presents certain limitations. Despite its high spatial resolution, its limited temporal resolution restricts the ability to capture rapid neural activity that may play a crucial role in visual processing. Future work could address these limitations by utilizing larger and more diverse fMRI datasets, incorporating higher temporal imaging techniques such as electroenccephalgrann (EEG) or electrocorticograply (ECoG), and exploring alternative stimulus representations beyond CLIP.
Additionally, our clustering approach might merge concepts within regions of CLIP space that have relatively uniform densities, potentially missing interesting SDCs. This issue could be addressed
9

---

Published as a conference paper at ICLR 2025
by adapting the hierarchical version of DBSCAN, or by using different criteria merge the cross- participant core points into clusters. The use of hierarchical DBSCAN could consider a range of epsilons in the same clustering run, allowing for both more and less dense clusters to be identified. This may lead to both distributed and localized clusters emerging as the hierarchical clustering proceeds.
Furthermore, we acknowledge that our method is not replacement for traditional hypothesis-dmie -rrive experiments. It will be important for future work test for alternative hypotheses for what might be driving responses in SDC clusters. Another interesting research direction is the application of this method to other sensory modalities beyond vision, such as auditory or somatosensory stimuli.
ACKNOWLEDGMENT
This research was supported by the Natural Sciences and Engineering Research Council of Canada (NSERC), the Social Sciences and Humanities Research Council (SSHRC), the Digital Research Alliance of Canada (alliancecan.ca) New Frontiers in Research Fund (Exploration), and the Canadian Institute for Advanced Research (CIFAR). Compute resources were graciously supplied by Prairies DRI and the Digital Research Alliance of Canada. Alona Fyshe holds Canada CIFAR Al Chair.
REFERENCES
E. J. Allen, G. St-Yves, Y. Wu, L.. Breedlove, J. S. Prince, L. T. Dowdle, M. Nau, B. Caron, Pestilli, Charest, B. Hutchinson, Naselaris, and K. Kay. A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence. Nature Neuroscience, 25(1), 2022. doi: 121133/441593-02-0009999
S. Bracci, M. letswaart, M. V. Peelen, and C. Cavina-Prateds Dissociable neural responses to hands and non-hand body parts in human left extrastriate visual cortex. Journal of Neurophysiology. 103 (0::3399-3399 June 2010. ISSN 0022-3077, 1522-1598. doi: 10.1152/j..00215520000
S. Bracci, A. Caramazza, and M. V. Peelen. Representational similarity of body parts in human occipitotemporal cortex. The Journal of Neuroscience, 15777)122777 Sept. 2015. ISSN 0270-6474, 1529-2401. doi: 10. 10.152/NEUROSSCI4699 -4....4156..11.115... 14.2015.
R. G. B. Campello, D. Moulavi, and Sander. Density-based clustering based on hicrarchical density estimates. In J. Pei, V. S. Tseng, L. Cao, H. Motoda, and G. Xu, editors. Advances in Knowledge Discovery and Data Mining, pages 160-172, Berlin, Heidelberg, 2013. Springer Berlin Heidelberg. ISBN 978-3-642-37456-2.
M. Carreiras, L. Carr, H. A. Barber, and A. Hernanezz. Where syntax meets math: Right intraparietal sulcus activation response to grammatical number agreement violations. Neurolmage, 49(2): 1741-1749, 2010. URL htt.:ttpp///bb p. un ceriittreer... de/ db/ journa natroimage49. htm #Ca HCarreirasCHHIC
P.-H. C. Chen, J. Chen, Y. Yeshurun, U. Hasson, J. Haxby, and P. J. Ramadge. A reduced-dimension fmri shared response model. In C. Cortes, N. Lawrence, D. Lee, M. Sugiyama, and R. Gar- nett, editors, Advances in Neural Information Processing Systems, volume 28. Curran Asso- ciates, Inc., 2015, URL wtt.tpppps/ppoeee..s..ooo.oooos.
files/paper/
S. Dehaene. Symbols and quantities in parietal cortex: elements of a mathematical theory of number representation and manipulation. 2007. URL https: swwanticscholar.oo ott.
Corpus 1100:1388330 33825044
S. Dehaene, M. Piazza, P. Pinel, and L. Cohen. Three parietal circuits for number processing. Cognitive Neuropsychology. 20(3-6):487-566 May 2003. ISSN 0264-3294, 1464-0627. doi: 10.10800264499904400029.
D. D. Dilks, J. B. Julian, A. M. Paunov, and N. Kanwisher. The occipital place area is causally and selectively involved in scene perception. Journal of Neuroscience. 3334)11331 2013. ISSN 0270-6474. doi: N0.523//NEUROSCI40811.11
URL httpss: ://www jneurosci
brre/contemt/334/133
10

---

Published as a conference paper at ICLR 2025
P. Downing, J. YH, M. Shuman, and N. Kanwisher. A cortical area selective for visual processing of the human body. Science (New York, N.Y.), 293:2470-3, 10 2001. doi: 10.1/26/sciene. 1063414.
R. Epstein and N. Kanwisher. cortical representation of the local visual environment. Nature, 392 (6676):598 601,,,1111 1998.
M. Ester, H.-P. Kriegel, J. Sander, and X. Xu. A density-based algorithm for discovering clusters in large spatial databases with noise. In Proceedings of the Second International Conference on Knowledge Discovery and Data Mining, KDD 96, page
AAAI Press, 1996.
D. Ganor- Y. Gliksman, S. Naparstek, G. Ifergane, and A. Henik. Damage to the intraparietal sulcus impairs magnitude representations of results of complex arithmetic problems. Neuroscience, 438:137-144, 2020. ISSN 0306-4522. doi: https://doi.coo org/10. 10...0016)))/ 1ttpg/...01//...noooooeeoe ne0o.
Y. Golland, P. Golland, S. Bentin, and R. Malach. Data-driven clustering reveals fundamental subdivision of the human cortex into two global systems. Neuropsychologia, 46(2):540-33 553,053335000 2008. ISSN 00283932. doi: 10.1016 neroooooooooyyoooia...000 10.000.
J. V. Haxby, J. S. Guntupalli, S. A. Nastase, and M. Feilong. Hyperalignment: Modeling shared information encoded in idiosyncratic cortical topographies. eLife, 9:e56601, jun 2020. ISSN 2050- 084X. doi: 10. (0.7554/eLife.5661)
URL https: //doi//do////ooooooooo heaedoi.orgggg// 10. 7554 10554/eLite fe. 56601
N. Jain, A. Wang, M. M. Henderson, R. Lin, J. S. Prince, M. J. Tarr, and L. Wehbe. Selectivity for food in human ventral visual cortex. Communications Biology, 6(1), 2023. doi: 10.1038/ $42003-023-04566.
N. Kanwisher, J. McDermott, and M. M. Chun. The fusiform face area: A module in human extrastriate cortex specialized for face perception. Journal of Neuroscience, 17(1)):402-4311 1997. ISSN 0270-6474. doi: 10. LL523/NEEURRSC111111000022..... jneurosci http://///cooee///////333333333
URL https: //www
M. Khosla, N. A. R. Murty, and N. Kanwisher. A highly selective response to food in human visual cortex revealed by hypothesis-free voxel decomposition. Current Biology, 32(19):4159-411 2022.
G. E. Koch, M. E. Libertus, J. A. Fiez, and M. N. Coutanche. Representations within the intraparietal sulcus distinguish numerical tasks and formats. Journal of Cognitive Neuroscience, 35(2):226-24) 240, Feb. 2023. ISSN 0889--99XX. doi: 10. 162/jocn_ a_01933.
T. Konkle and A. Oliva. A real-world size organization of object responses in occipitotemporal cortex. Neuron, (416))111441111 June 2012. ISSN 08966273. doi: (e00066/.eeuro..211222222 201...04..........
N. Kriegeskorte, W. Simmons, Bellgowan, and C. Baker. Circular analysis in systems neuroscience: The dangers of double dipping. Nature neuroscience, 12:535 40, 05 2009.
D. Lashkari, E. Vul, N. Kanwisher, and P. Golland. Discovering structure in the space of fmri selectivity profiles. Neurolmage, 50(3): Apr. 2010. ISSN 10538119, doi: 10.1016/j. neuroimage.2009.11 12.166.
T.-Y. Lin, M. Maire, S. Belongie, L. Bourdev, R. Girshick, J. Hays, P. Perona, D. Ramanan, C. L. Zitnick, and P. Dollár. Microsoft coco: Common objects context, 2014. cite arxi::1405.03122ommm 10..m2cmmmmnt: 1) updated annotation pipeline description and figures; 2) added new section describing datasets splits; updated author list.
Y. Liu, Y. Ma. W. Zhou, G. Zhu, and N. Zheng. Brainclip:
Bridging brain and visual-linguistic representation via clip for generic natural visual stimulus decoding, 2023. URL https:// 12977
A. Luo, M. Henderson, L. Wehbe, and M. Tarr. Brain diffusion for visual explo- ration: Cortical discovery using large scale generative models. In A. Oh, T. Naumann, A. Globerson, K. Saenko, M. Hardt, and Levine, editors, Advances in Neural In- formation Processing Systems, volume 36, pages 75740-75781. Curran Associates, Inc., n http://ppper.lllle/ppppp///////a ffBeffaaaaaaanaeeeeeeeroooeeeeer
2023a. URL http:///ooo/////oooeeoeeoeeeoono

---

Published as a conference paper at ICLR 2025
A. F. Luo, M. M. Henderson, M. J. Tarr, and L. Wehbe. Brainscuba: Fine-grained natural language captions visual contex selectivity. CoRR, abs/2310.04420, 2023b. doi: 10.48550/ARXIV.33100 04420. hRttps:://doot http://o......g.oooo/oo//o 48550/ http://aiiii...... 3
E. Maguire. The retrosplenial contribution human navigation: A review of lesion and neuroimaging findings. Scandinavian Journal of Psychology, 42(3):225-238, 2001. doi: https://doi.or// 1111 1467-9450.00220 URL BERLaatcas//ooannee.aeeaa.eeeeee wttp://cwcc.oooo/////o/////ccc/a 1467-9450. 60233
A. Nieder and S. Dehaene. Representation of number in the brain. Annual review of neuroscience, 32:185-208, 04 2009, doi: 1.11////urrre.neeurg 1500. 135550.
S. Norman-Haignere N. G. Kanwisher, and J. H. McDermott, Distinct cortical pathways for music and speech revealed by hypothesis-free voxel decomposition. Neuron, 88(6):1211-1299 Dec. 2015. ISSN 08966273, doi: 10. wetp:...nrooo.....111111
T. Orlov, T. R. Makin, and E. Zohary. Topographic representation of the human body in the occipitotemporal cortex. Neuron, 68(3):586-666 600, Nov. 2010. ISSN 08966273. doi: 10.1016/j. ne0.0..2010.09.033.
I. M. Pennock, C. Racey, E. J. Allen, Y. Wu, T. Naselaris, K. N. Kay, A. Franklin, and J. M. Bosten. Color-biased regions in the ventral visual pathway are food selective. Current Biology. 33(1): 134 146, 2023.
J. S. Prince, Charest, J. W. Kurzawski, J. A. Pyles, J. Tarr, and K. N. Kay. Improving the accuracy of single-trial fmri response estimates using gimsingle. eLife, 11:e77599, nov 2022. ISSN 2050-084X. doi: 10.7554/eLife.77590 775599.
Pérez-Ortega, A. Akrouh, and R. Yuste. Stimulus encoding by specific inactivation of cortical neurons. Nature Communications, 15(1):3192, Apr. 2024. ISSN 2041-1723. doi: 10. 1038/
14446-024--7
A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, Mishkin, Clark, G. Krueger, and Sutskever. Learning transferable visual models from natural language supervision. CoRR, abs/2103.00020, 2021.
G. Sarch, M. Tarr, K. Fragkiadaki, and L. Wehbe. Brain dissection: fmri--raines net- works reveal spatial selectivity in the processing of natural images. In A. Oh, T. Nau- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neural Information Processing Systems, volume 36, page 46285-46283 Curran Associates, Inc., 2023. URL
www.:iiinnnn.uuuuuuuu.u www. cw/paperrcppppp.peppppppppp files//aperr2023/ File/ 2t04:/98864422220002222222222223 Sewapper-Cooffeeeeeee pdf
A. Stigliani, K. S. Weiner, and K. Grill-Spector: Temporal processing capacity in high-level visual cortex is domain specific. Journal of Neuroscience, 0270-6474, 1529-2401. doi: N01523/NEUROSSC148222111
Sept. 2015. ISSN
B. T. Thomas Yeo, F. M. Krienen, J. Sepulcre, M. R. Sabuncu, D. Lashkari, M. Hollinshead, J. L. Roffmman, J. W. Smoller. Zöllei, R. Polimeni, B. Fischl, H. Liu, and R. L. Buckner. The organization of the human cerebral cortex estimated by intrinsic functional connectivity. Journal of Neurophysiology, 106(3): 125--115,, Sept. 2011. ISSN 0022-3077, 1522- doi: N0.11522jn.00338.2211
A. van den Oord, Y. Li, and O. Vinyals. Representation learning with contrastive predictive coding. CoRR, 0obs/1807.00 .03748, 2018.
K. S. Weiner and K. Grill-Spector, Not one extrastriate body area: Using anatomical landmarks, hmt+, and visual field maps to parcellate limb-selective activations in human lateral occipitotemporal cortex. Neurolmage, 56(4):2183-2199) June 2011. I. ISSN 105381 19. doi: 10.10 neuroimmee........... 2011.03.044
12

---

Published as a conference paper at ICLR 2025
A
SUPPLEMENTARY MATERIAL
A.1 DERIVING SDC CLUSTERS
b
C
linear decoder
minlleighhours
10
DBSCAN
patecpertt
(0.5, 0.55, 0.66)
d
cluster centroid
W
X
k
W
Y k Test
512
Test
f
distance cosine W L AVG
Test
0.1
0.3
0.5
0.8
1.0
1.12
15
1.76
1.92
g
positive
negative
Figure 10: Deriving SDC clusters. (a): The participant-specifii linear decoders derived Figure
A.2 (b): Our modified DBSCAN clustering procedure is applied to the linear decoders (See Figure T
for details). (c): The DBSCAN algorithm discovers cluster centroids in figh-dimensional space. (d):
The 512-dimensional representations from the previous step averaged over voxels and participants
to derive a cluster centroid for each cluster derived from DBSCAN. We visualize the cluster centroid for the first DBSCAN cluster. (e): The linear decoders from (a) are applied data per-participant. Each participant saw images multiple times, so the matrix of predicted CLIP embeddings YAa.. is averaged over these repetitions and all linear decoding matrices are stacked (across participants) give Y,AVG (f): Cosine distance is calculated between the cluster centroids (e) and the brain-derived CLIP embeddings (f). (g): The images most associated with the cluster centroids (positive images) and most negatively associated with the cluster centroids (negative images) are identified. Positive negative images for the SDC cluster pictured here appears to correspond to global vertical/horizontal orientation in the associated images. (h): Color-coded participant-specine voxel clusters are displayed on flatmap of the brain's 's cortical surface in common fsaverage space (overlapping areas are displayed in white). Regions of interest labels are highlighted the flatmap image in white outlines. For the specified cluster (d), whose positive negative images are associated with orientation, the flatmap indicates bilateral shared voxel clusters in early visual cortex.
held-out fMRI
A.2 DECODING CLIP REPRESENTATIONS
The decoding model is trained for 5000 iterations (29 to 45 epochs depending the participant's training set size) with the Adam optimizer, a batch size of 128, and fixed learning rate le Data augmentation is applied to help slow overfitting by adding random noise training samples
13

---

Published as a conference paper at ICLR 2025
training images
Train
CLIP
Contrastive Loss
Train
Linear trainable
N
512
[MRI responses
Figure 11:
Decoding CLIP representations from the brain. We create CLIP representations by passing NSD training set stimuli to CLIP (frozen). The NSD team showed these same images to human participants during fMRI scans. We train linear decoder with contrastive loss (Eq. 2 to predict CLIP embeddings from the fMRI responses to the corresponding images. Black arrows represent the flow of data through the procedure and dashed red lines represent gradient updates used to train the model.
2; + 2; + z sampled from a normal distribution z ~ N/0,, o²) where the noise standard deviation is a ppper-rrrameter.. We selected based on performance on the validation set. The exact number of voxels input to the model was [17883, 18358, 13476, 11899, 17693, 18692, 9608, 6772) for subjects through 8. We compare our contrastive decoder to baseline ridge regression model trained on the same data. We used grid search to select the best ridge regularization parameter À (0.1, 10, 100, 1000, 10000, 100000) using the validation data. The optimal was 10000 for participants 1, 2, 5, 6, and 1000 for participants 3, 3,4,7 7,
0.03 and 0.1 in our implementation. Hyper- parametrtss were
A.3 COMPUTE RESOURCES
The decoder models were trained on an NVIDIA GeForce RTX 2060. Training time was approx- imately 2 minutes for each model. The total training time for 50 models for each of the 8 NSD participants was about 13 hours. The DBSCAN clustering algorithm uses minimal CPU resources and executes within few minutes.
A.4 REGION OF INTEREST IDENTIFICATION
In order to localize shared voxel clusters in our visualizations, overlaid region of interest bound- aries onto the flatmap visualizations Freesurfer's fsaverage space. For functional ROIs, we retraced the ROIs given in the NSD dataset during the functional localization experiments (fLoc, (Stigliani et al. (2015)..
A.5
TOP-K ACCURACY FOR CLIP DECODING
See Figure 12
A.6 VOXEL NORMALIZATION
Brain responses voxel normalized to have mean and unit standard deviation within each scanning session. As a result, we allow brain response values to be positive or negative. In order to study both positive and negative aspects of stimuli-driven brain responses, this is
14

---

Published as a conference paper at ICLR 2025
Commooiie
Ridge Bazcline
Chance
Contrnaiive parteipanes)
et particppnts)
top-k top-k accuracy
60
50-
40
30
10
50
Figure 12: Top-k accuracy for CLIP decoding using the proposed contrastive decoder and ridge regression baseline. The contrastive decoder implementation outperforms the ridge regression baseline across various values of k. Chance performance is given in green. Accuracy calculated on held-out data. Bars indicate the standard error (SE) across the 8 NSD participants. Additionally, we include comparison Liu et al. (2023) who report the mean top-k accuracy across 4 subjects (1, 2, 5, 7) using brain responses to 982 test images (dashed red line). We average the results of our own contrastive decoder across these participants for a fair comparison (dashed blue line). Despite slightly larger test set, our contrastive decoder has higher accuracy for all reported values of k.
Positive
Negative
Positive
Negative
Figure 14: Cluster 107 (E = 0.569). Positive
Figure 13: Cluster 110 (E 0.563). Positive images display crowds, while negative images are individuals. Voxel clusters are bilateral EBA.
images show people jumping leaping, while negative images are of people standing still or sitting. Voxel clusters are in and around left hemisphere EBA.
necessary preprocessing step as the beta weights in NSD are calculated with respect to a baseline gray background which is not zero-centered (Allen et al. 2022)
A.7 PLACES
Cluster 115 (Figure 17) strongly overlaps with PPA and displays outdoor scenes with consistent vegetation for the positive images. The negative images show indoor scenes where human-made objects are prominent, with distinct lack of vegetation.
Cluster (Figure 2T displays indoor scenes for the positive representative images. This cluster mostly localized in OPA with some voxels in PPA. The positive images depict cluttered indoor scenes such as kitchens, work spaces, and living rooms. There is typically a flat surface such desk, countertop, or table the focal point of the image. The negative images depict outdoor scenes with animals.
Cluster 122 (Figure 22) overlaps strongly with PPA, OPA. and RSC. The positive representative images usually display an urban area where the camera is looking forward down a long path or road. In contrast, the negative images contain many close-up images of objects on tables where
15

---

Published as a conference paper at ICLR 2025
Positive
Negative
Posilive
Negalive
Figure 16: Cluster 87 (E 0.596). Positive im- ages show people with bent legs or animals laying down. Negative images primarily show people standing. Voxel clusters are in bilateral EBA.
Figure 15: Cluster 64 (e = 0.657). Food-related images with shared voxel clusters in anterior re- gions of the brain.
Posltive
Negative
Figure 17: Cluster 115 (8 = 0.555). Positive images display scenes with plants and foliage, while negative images show human-made objects. Voxel clusters are in bilateral PPA.
the surrounding environment is not visible. This suggests that this cluster may be related to scene geometry, navigation, or the reachability of locations a scene (Epstein and Kanwisher 1998 Dilks
et al. 2013 Maguire 2001
A.8 BODIES (CONTINUED)
In Section 4.3 we identified various SDCs relating to the presence of bodies in images. We expand on a few further observations here. Cluster 110 (Figure 13) shows groups of three or more people in the positive images, with a strong focus on an individual person, animal, or object in the negative images. Cluster 107 (Figure 14) appears to be related to full-body leaping motions, with the negative images showing people sitting or standing still. In contrast, cluster 87 shows mixture of images of people who are crouched sitting, along with images of pet cats and dogs laying down.
A.9 FOOD
Cluster 64 (Figure 15) is shared across seven participants and is localized to frontal brain regions. Voxels are located in the orbital sulci, as well as the boundary between the triangular part of the inferior frontal gyrus and the inferior frontal sulcus. We noted that the positive images to contain images of food on plates, as well as few non-food images (bear, skiing, playing baseball). We had trouble discerning a strong core concept in the negatively associated images, aside from a distinct lack of food.
16

---

Published as a conference paper at ICLR 2025
Positive
Negalive
Negative
Figure 18: Cluster 75 (E 0.631). Positive images contain textiles, bedding, and other soft things. Negative images show hard objects. Voxel clusters are primarily in the right hemisphere be- tween EBA and FFA.
Figure 19: Cluster 119 0.534). Positive images display contrast between light source and a dark environment. Negative images show uniform ambient lighting. Voxel clusters are pri- marily in bilateral RSC and PPA.
A.10 SOFT HARD CLUSTER
shows images with strong focus on clothing, bedding, and other textiles. The negative images show trains, and concrete-dominant architecture. The voxels in this cluster are mostly found the right hemisphere bordering the area between FFA and EBA.
Cluster 75 (Figure 18
A.11 LIGHTING-RELATED CLUSTER
Cluster 119 (Figure 19) includes voxels in RSC and PPA. The positive images depict scenes with high contrast in lighting. There typically a dark environment with bright light window that is partially illuminating the room. Conversely, the negative images depict close-up pictures of objects with uniform ambient lighting. This strongly suggests that this cluster voxels responds to images that display a high contrast in lighting.
A.12 REPRESENTATIVE WORD CLOUDS
In Figure 20] we present word clouds containing the most frequent words in the representative captions for each image. In most cases the word clouds support our interpretation of each underlying cluster concept. For example the word cloud for the lighting cluster (Figure [9) containing the words lit, lamp, sun, illuminated, dark, dimly, brightly, all of which are indicative of lighting conditions. Similarly, the repeated elements cluster (Figure 8) is well-characterized by words like group, together, formation, squadron, herd, which emphasize the concept of repetition or grouping.
In some cases, while the word clouds provide partial support for our descriptions, the thematic patterns become more apparent when examining the representative images. For example the positive representative words for the hands cluster (Figure contains action-oriented words like as cutting and slicing, and the legs cluster (Figure 4) features words such riding and running. However it is difficult to identify the overall theme of hands or legs without the representative images.
The word clouds for the food clusters offer additional insights. The food and color cluster (Figure 3) is strongly indicative of favoring the color versus grayscale concept. In contrast, the anterior food cluster (Figure 1.5) has strong association to food-related terms like food, eating, plate, avocado, and feeding.
A.13 INTERACTIVE RESULTS VIEWER
Alongside this paper, release interactive web application that allows for the exploration of the full hierarchy of results. A preview of our viewer is displayed in figure 23 The viewer can be accessed at the following URL: Syshhelab.githee ttob. fo/bra n-viewer
17

---

Published as a conference paper at ICLR 2025
POSITIVE
NEGATIVE
POSITIVE
NEGATIVE
LnS knife if
horse
beach
benches oe Li
hholdin
man
Fig. Faces
pizze a
per son
animal
ganduna grouD
Fig. 6 Horzontal
AITIO AWOLayiT
Fig. Food and colour
yeer OW
white photo
Fig.7 7 Vertical
Bar KL0g Ne
Shes
F SOU
Freggrea
for Torful
black white
grass
gr ass
15 Anterior food
Two food gifafiee 1ood
Carpet Auggage
17 Nature
fence
green EE
00
Foo sitting
compopcomputer
plant
man road "SS wsstrees
cat table
counter Wooden
Fig 4. Legs
Fig. 21 Indoor
skateboard
kitchen
SMPTAAEE iiing
cha
scenes
table Widdow
Fitting knife
horse
street build ding
ConDUter Taptop
Hends
pizza=
Monani Rer son
P
animal
Ranounaagr Baggr rou
Outdoor scenes
carFoad
FOUSE
Fig. 14 Bodies in Motion
tennis Blayer
tumpong Ternis
skateboar del
sitting
woman Tra
Fig. 19 Lighting
window
Eviing HEoom
1pht dark
50 pu banch. picoic table
onnar boat
Secatei pers
celf phone
16 Sitting
chair
dyingdog
group
18 Softness
b blanket
Sheet 80
Kitchen sitting cal Saccuee
umbrella
bed
Lying typ
cat
GOunit
people
front kiir posing F Girt 100 snow
black
Fig. 13 Crowds
Fig. 9 Closelter
group
D arrge
white
Teshant
Siign
ski standing
Fig. Repeated elements
Baaan fire hydrant Sebied
srtting
gruppp
Figure 20: Word clouds for all SDC clusters. The positive and negative representative words are displayed side-by-side along with subset of the representative images. The text to the left identifies the corresponding flatmap figure and our interpretation of the SDC.
Positive
Negative
Positive
Negative
Figure 21: Cluster 79 (E 0.607). Positive im- ages show indoor scenes with desks and tables. Negative images show outdoor seenes with an- imals. Voxel clusters are primarily in bilateral OPA and PPA.
Figure 22: Cluster 122 (E = 0.528). Positive
images show scenes with large open areas. Neg- ative images are close-ups of objects that have no scene geometry. Clusters span bilateral RSC, OPA, and PPA.
18

---

Published as a conference paper at ICLR 2025
Figure 23: Preview of our interactive viewer application. The user can select minNeighbors whether to display the positive or negative representative images, and the cluster ID. The cortical surface can be toggled between flattened, inflated, pial, or white matter surface meshes with the controls in the bottom left corner.
19

---

Published as a conference paper at ICLR 2025
A.14 14
PROJECT CODE
Code to reproduce this work can be found at the following URL: https:// hatps://gitthnn /gigggu..hh..o/// fyshe wtt/:/onttaaattteeeeeee---iii-ni
20


Reviews:
[{"review_id": "n7604Ru5IE", "reviewer": "Reviewer_oMEW", "summary": "This study aims to identify brain areas that respond to specific visual concepts, aiming to expand our understanding of functional localization beyond known areas like the fusiform face area. Using a model based on CLIP embeddings, the authors map brain responses to clusters of shared visual concepts, called \"Shared Decodable Concepts\" (SDCs), across multiple participants. By applying a modified DBSCAN clustering algorithm, they reveal consistent brain regions linked to both previously known (e.g., faces, places) and novel visual concepts (e.g., specific body parts, numerosity, perspective). This method particularly allows data analysis within each participant’s unique brain structure, avoiding issues with traditional alignment methods.", "questions": "The authors frame their study in the context of previous work on identifying functionally localized brain regions. However, it is unclear how their methodology would specifically highlight concepts confined to a few visual areas versus those spread across the brain. Instead, the approach appears to identify concepts that are easily decodable from brain signals across participants, regardless of whether decoding relies on signals from a small localized area or broader, distributed regions.", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 2, "strengths": "- The discovered novel selectivities observed in the study are quite interesting and potentially good hypotheses for future brain imaging studies \n- The perspective that both active and suppressed neural responses are useful for identifying brain regions tied to distinct visual concepts is somewhat unique and not something I've seen in prior studies \n- This method circumvents the need to align subjects data, while still being able to extract shared representational structure across participants", "weaknesses": "- More methodological details are needed. More motivation behind why DBSCAN clustering used over other clustering algorithms? How do the authors select the epsilon which ultimately determines the number of clusters? \n- The authors should acknowledge that there is a lot of human bias involved in interpreting the concepts based on the activated images. For e.g. in 4.1, the authors interpret the negative representative images to indicate images that lack clearly visible faces. However, many other interpretations are also consistent with these sets of images (e.g. they all contain scenes). Similarly Cluster 2 in Fig. 5 could mean something other than bodies too (implied motion?). The subjectivity involved in interpreting the clusters makes me less enthusiastic about the study. Automating this interpretability analyses in future work could be interesting. \n- To better understand the strengths of the proposed methodology, it would also be helpful to compare it against natural candidates for understanding shared representational structure across participants, like Canonical Correlation analysis", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "SbIWI3HE1Y", "reviewer": "Reviewer_FqNe", "summary": "The authors propose an approach for detecting \"Shared Decodable Concepts\" in human visual fMRI data: clusters in CLIP space that are decodable from common sets of voxels across multiple participants. These clusters are identified using a modified implementation of the DBSCAN clustering algorithm, which identifies clusters in high-dim spaces. In particular, the algorithm operates over vectors of linear decoding weights that map between voxels and CLIP latents. Each weight vector is treated as a brain-decodable concept for a given voxel, such that cosine distance between the weight vectors corresponding to different voxels from different subjects can be used to index the similarity of the concepts encoded therein. After clustering, the authors interpret the SDCs by inspecting a set of \"positive\" representative images (and words) that are associated with the centroid of each one, and by further identifying groups of \"negative\" images (and words) that are represented in a maximally different manner. These analyses enable the authors to identify many well-studied aspects of visual cortical organization, such as areas that are tuned to face, body, scene, and word content, and to identify some less-well-studied features such as crowds and lighting. These observations may form the basis for future hypothesis-driven studies that seek to further probe the nuances of the representations of these visual concepts.", "questions": "- One main question I am left with is: how many of these proposed clusters contain novel interpretable forms of feature tuning? It is difficult to know without a much more systematic attempt to cite relevant literature on visual category selectivity. \n- It remains unclear to me whether cross-subject analysis is required to derive these forms of insight, as discussed above. \n- Another key question relates to the issue of bias mentioned above. It would be straightforward to attempt to replicate some of these analyses using BOLD5000, THINGS, or other large datasets to address the possibility of bias that arises from the NSD stimuli. Another critical issue is whether relying on the CLIP feature space unavoidably biases the outcomes of analyses toward semantically interpretable outcomes. In other words, does the use of CLIP cause us to \"overfit our understanding\" of these brain areas to a feature space that was constrained to be language-aligned? I am not positive how I would go about addressing this issue, but at minimum I would appreciate if the authors attempted to discuss it. \n- Given that orientation tuning is a hallmark of V1 representations, why did V1 not show up in the corresponding cluster(s), only V2/V3?\n\nOverall, given the extent of concerns listed above, I am unable to rate the paper higher than a 3 currently. In principle, I am willing to raise my review score if the authors are able to better situate this work in the context of other literature, and more clearly pinpoint the specific novel findings/predictions that differentiate it from other work that has used CLIP to try to understand NSD.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The idea of clustering different subjects' parameter/weight vectors as a means of studying what is common (or different) across individuals' brains is a promising direction that merits more attention and exploration in the field. It's a clever way to overcome the challenges of aligning different brains, while respecting the nuances of individuals' representational signatures. \n- Beyond the well-studied forms of category selectivity that pop out in the SDC analysis, some of the observations are new and interesting (e.g. a \"softness\" cluster). Identifying these signatures was only possible because of the sophisticated overall methodological approach applied to a rich dataset. These provide provocative directions for future research, to be sure they are actually replicable signatures across datasets rather than clusters that are biased by the specifics of NSD and/or the use of CLIP. \n- The authors make use of the richest possible data (NSD) available, and I appreciate the careful treatment of noise ceiling-related issues in the paper. It's also important that the authors ran a control comparing CLIP decoding to ridge regression. \n- It's useful that this work provides further empirical perspective on the hotly debated topic of whether food selectivity is separable from color representation in the ventral stream. \n- It's powerful to provide clear visual intuition of both the positive and negative examples that correspond to each cluster, as a way to bolster both the interpretability and the hypothesis space that each SDC provides. \n- I deeply appreciate that the authors discussed the possibility of bias in the conclusion, which might arise from the use of NSD stimuli, and which might arise from the use of CLIP.", "weaknesses": "- In general the paper does a poor job citing other relevant literature, missing some critical papers on category selectivity in the visual system, as well as on recent efforts to use CLIP models for data-driven interpretability of visual feature tuning. Given the topic of the paper, the fact that it has only 20 citations total is insufficient to cover recent work in this area. For example, the authors should connect their observations of body-related selectivity to other works such as: https://www.nature.com/articles/s41467-020-16846-w, https://www.nature.com/articles/s42256-023-00753-y, and perhaps others.  Citing few works beyond the \"original\" papers identifying FFA/PPA etc makes it hard to understand how many of the findings are novel versus replicates of other studies. I was also surprised that the authors did not cite/discuss the BrainDIVE paper (https://arxiv.org/pdf/2306.03089) and any other works that use NSD and CLIP to propose new kinds of representational distinctions and signatures in the visual system. Furthermore, Stan Dehaene has relevant work on numerosity representations in the brain that should be cited/discussed. The discussion of food/color representations is missing a key citation from Pennock et al (https://www.cell.com/current-biology/pdf/S0960-9822(22)01893-0.pdf)... and so forth. I'd also recommend discussing existing methods of functional / representational alignment that do not required projecting individuals into a shared anatomical space, such as hyperalignment (Haxby) and shared response modeling (Chen). Why would simpler methods like these be insufficient for this problem of revealing shared dimensions and visualizing the images that project strongly/weakly onto them? I believe that there's extra \"juice\" to be gained from the CLIP-based approach, it would just help if this historical context were touched on somewhere in the paper. \n\n- There are many ways to achieve high distance from a given cluster centroid in high dimensional space. I understand the reason for looking at the most- and least-associated images with each SDC cluster, but it's not clear that being atypical in a clustering sense implies that those images would actually have a suppressive impact on the voxels' or regions' response profiles. The results here provide clear testable hypotheses, but the impact of the work is limited without further data collection to validate/falsify the positive-negative axes the authors propose. \n- The introduction should convey why combining data across participants is a necessary step to discover new forms of category selectivity - this is not obvious. Would a straightforward clustering algorithm applied to the voxel tuning within individual subjects reveal many of these same clusters? The authors do not present analyses that clearly show that cross-subject analysis is necessary to uncover particular forms of interpretable latent dimensions. \n\n- The clarity could have been improved by streamlining the extensive methods description that occurs before any main results are presented, leaving the nitty-gritty details for the supplement. Sections 2 and 3 place a high working memory burden on the reader - reducing this could help a lot with readability. \n\n- This might seem like a minor/pedantic point, but the authors' use of the word \"semantic\" to describe feature tuning in high-level visual cortex is a bit fraught - even if different regions have selectivities that seam meaningful to the human eye, that does not mean that the neurons themselves necessarily compute semantically interpretable features. (Furthermore, language fMRI researchers likely also have a very different conception of what \"semantically tuned patches of cortex\" means, compared to vision researchers.) I would recommend that the authors reconsider wordings such as \"semantic concepts that drive activity\" or \"semantically-tuned patches\" throughout the paper, in favor of language that describes forms of \"visual feature tuning\" that may be semantically interpretable, post-hoc, without implying that the underlying cells themselves \"know\" what semantics are.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ACWdU6Buva", "reviewer": "Reviewer_LDPq", "summary": "This paper proposes a new method to discover semantically interpretable fine-grained functional organization of fMRI responses that are consistent across participants. It then applies this method to the NSD data form Allen et al, reporting both established functional responses (FFA etc and responses claimed to be novel (e.g. for particular body parts, numerosity, etc).\n\nI recommend acceptance of the paper because the method proposed is useful, the writeup is clear, and the analyses appear solid.", "questions": "Recommendations to improve the paper:\n\n1. At least cite some of the relevant prior literature as discussed above.\n2. It would be useful if the authors could clarify whether the \"suppressed\" responses are simply lower than those to other stimuli, or whether they are actually lower than a minimal baseline (e.g. the response when no stimulus is presented). In this reviewer's opinion \"suppression\" should refer to a response below the no-stimulus baseline, and it is not clear that is the case here.\n3. The claim that food-related voxel clusters \" span bilateral FFA, V4, and PPA.\" doesn't make sense given the usual definition of FFA and PPA as face and place selective. Presumably what is mean here is that food-related voxels are found near the FFA and PPA.\n4. The authors should show how well the brain data reproduces CLIP embeddings (and how this changes over training). That is, show how similar Y_hat is to Y_CLIP. \n5. How does using the CLIP dimension (w_i) improve on just using the neural response values (x_i) in this method? \n6. Perhaps comment on whether these results tell us more about the brain or about CLIP.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "Strengths of the paper are that it tackles an important problem, applies it to a beautiful data set, and reports interpretable results that either serve to establish the validity of the method (via replication of established findings) or report intriguing new results. Another strength is the emphasis on the lowest responses of functional clusters, something that is widely appreciated in the field but rarely if ever explicitly discussed. A third strength is the care that was taken to re-calculate the per-voxel noise ceiling estimates on training data only, a small but important point.", "weaknesses": "The main weaknesses of the paper concern the failure to situate the paper in the context of relevant prior work. Some of the results are described as novel without reference to related findings published previously, For example, the claim of selectivity for different aspects of the body should acknowledge a large prior literature on this topic, e.g.: \nOrlov, T., Makin, T. R., & Zohary, E. (2010). Topographic representation of the human body in the occipitotemporal cortex. Neuron, 68(3), 586-600.\nWeiner, K. S., & Grill-Spector, K. (2011). Not one extrastriate body area: using anatomical landmarks, hMT+, and visual field maps to parcellate limb-selective activations in human lateral occipitotemporal cortex. Neuroimage, 56(4), 2183-2199.\nBracci, S., Caramazza, A., & Peelen, M. V. (2015). Representational similarity of body parts in human occipitotemporal cortex. Journal of Neuroscience, 35(38), 12977-12985.\nBracci, S., Ietswaart, M., Peelen, M. V., & Cavina-Pratesi, C. (2010). Dissociable neural responses to hands and non-hand body parts in human left extrastriate visual cortex. Journal of neurophysiology, 103(6), 3389-3397.\nbig-small close/far this is the ref:\nKonkle, T., & Oliva, A. (2012). A real-world size organization of object responses in occipitotemporal cortex. Neuron, 74(6), 1114-1124.\nAnd concerning the proposed method, data-driven analyses of fMRI data especially using clustering have been reported for at least 15 years, and this should be at least acknowledged, e.g.:\nGolland, Y., Golland, P., Bentin, S., & Malach, R. (2008). Data-driven clustering reveals a fundamental subdivision of the human cortex into two global systems. Neuropsychologia, 46(2), 540-553.\nLashkari, D., Vul, E., Kanwisher, N., & Golland, P. (2010). Discovering structure in the space of fMRI selectivity profiles. Neuroimage, 50(3), 1085-1098.\nYeo, B. T., Krienen, F. M., Sepulcre, J., Sabuncu, M. R., Lashkari, D., Hollinshead, M., ... & Buckner, R. L. (2011). The organization of the human cerebral cortex estimated by intrinsic functional connectivity. Journal of neurophysiology.\nand contrasted with alternative data-driven methods using ICA, NMF, etc, e.g.:\nNorman-Haignere, S., Kanwisher, N. G., & McDermott, J. H. (2015). Distinct cortical pathways for music and speech revealed by hypothesis-free voxel decomposition. Neuron, 88(6), 1281-1296.\nKhosla, M., Murty, N. A. R., & Kanwisher, N. (2022). A highly selective response to food in human visual cortex revealed by hypothesis-free voxel decomposition. Current Biology, 32(19), 4159-4171.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9lkrr0fiKe", "reviewer": "Reviewer_UQYg", "summary": "This paper presents a novel technique for discovering and interpreting functionally specialized clusters in the brain. Using the Natural Scenes Dataset, the authors train linear decoders to map single voxel activities onto CLIP embedding vectors for the respective images seen in the neuroimaging experiment. Then, the authors leverage the similarity of voxel->CLIP prediction weights to perform clustering. Specifically, the authors develop a variant of the DBSCAN algorithm that leverages across-subject consistency in order to find the most consistent and semantically meaningful components. The authors present the results for several components, showing the spatial topography as well as the most strongly and weakly activating images. The authors interpret the components in light of both their spatial topography, and the content that appears present in the strong and weak activating images.", "questions": "- Can you explain this sentence: \"We emphasize that there is no constraint that voxels within a shared visual semantic cluster be spatially adjacent in the brain\". \n\n- How does this paper go above and beyond the findings of other recent works exploring functional selectivity using the Natural Scenes Dataset?\n\n\n- Can you provide some intuition for why the contrastive fitting of voxel -> clip latents outperforms ridge regression?", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- Interesting method\n- The use of a contrastive loss in the fitting of clip latents from voxel activities is particularly interesting\n- The modification of DBSCAN is elegant, and provides an intriguing way of aggregating information across subjects\n- Good figures", "weaknesses": "- The early vision components (horizontal and straight lines) activate parts of V1-V3 that represent the periphery (if the authors are unaware: on the flat map, the center of the strips of V1-V3 corresponds to foveal bias, and moving outward corresponds to increasing peripheral eccentricity). This isn't discussed but seems important. \n\n- The results are very qualitative. The paper could be strengthened by making some objective quantifications of the content of different components, rather than just \"reading the tea leaves\". \n\n- This paper joins with a serious of related papers exploring visual components (e.g. Khosla et. al 2022; BrainDIVE, Luo et. al 2023 NeurIPS; BrainSCUBA, Luo et. al 2024 ICLR), to name a few. I'm not sure if there is any single result here that is particularly novel. This makes the paper less compelling. Additionally, the method is not compared to other methods, so it is unclear whether it holds any advantages. \n\n- The paper is very light on references. In my opinion, having 20 references for a paper on such a popular topic is a bit of a disservice to the field. \n\n- This sentence is confusing: \"We emphasize that there is no constraint that voxels within a shared visual semantic cluster be spatially adjacent in the brain, yet during visualization we consistently find contiguous patches both within and across participants\". Please explain how this can be. The algorithm appears spatial, in that core points are defined as having at least minNeighbors points within their epsilon (spatial) neighborhood. Additionally, the epsilon expansion appears to be highly spatial in nature. If it is truly the case that there is no spatial constraint, it would be a mistake to simply call the algorithm a DBSCAN variant -- it would be better to say that it is inspired by DBSCAN, then give it a new name to avoid confusion. Actually, this is probably a good idea regardless.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)