In Silico Mapping of Visual Categorical Selectivity Across the Whole Brain
Authors: Ethan Hwang
Hossein Adeli
Wenxuan Guo
Andrew Luo
Nikolaus Kriegeskorte
Venue: NeurIPS 2025 poster ()
Keywords: neural encoding
brain mapping
transformers
generative diffusion model
Primary area: neuroscience_and_cognitive_science
TL;DR: We built a transformer-based model to predict brain activity across the whole brain from visual input, then use this model to label the categoriacal selectivity of areas beyond the visual cortex to better understand higher-order visual processing.

Abstract: A fine-grained account of functional selectivity in the cortex is essential for understanding how visual information is processed and represented in the brain. Classical studies using designed experiments have identified multiple category-selective regions; however, these approaches rely on preconceived hypotheses about categories. Subsequent data-driven discovery methods have sought to address this limitation but are often limited by simple, typically linear encoding models. We propose an in silico approach for data-driven discovery of novel category-selectivity hypotheses based on an encoder–decoder transformer model. The architecture incorporates a brain-region to image-feature cross-attention mechanism, enabling nonlinear mappings between high-dimensional deep network features and semantic patterns encoded in the brain activity. We further introduce a method to characterize the selectivity of individual parcels by leveraging diffusion-based image generative models and large-scale datasets to synthesize and select images that maximally activate each parcel. Our approach reveals regions with complex, compositional selectivity involving diverse semantic concepts, which we validate in silico both within and across subjects. Using a brain encoder as a “digital twin” offers a powerful, data-driven framework for generating and testing hypotheses about visual selectivity in the human brain—hypotheses that can guide future fMRI experiments. Our code is available at: https://kriegeskorte-lab.github.io/in-silico-mapping-web/.

In Silico Mapping of Visual Categorical Selectivity Across the Whole Brain
Ethan Hwang
Hossein Adeli
Zuckerman Mind Brain Behavior Institute
Zuckerman Mind Brain Behavior Institute
Columbia University
Columbia University ha2366@columbia. edu
@h2976@columbia.cm edu
Wenxuan Guo Zuckerman Mind Brain Behavior Institute Columbia University
Andrew Luo University of Hong Kong aluo@hku. hk
wg23618columbia.c edu
Nikolaus Kriegeskorte Zuckerman Mind Brain Behavior Institute Columbia University
nk2765@columbia.. edu
Abstract
A fine-grained account of functional selectivity in the cortex is essential for under- standing how visual information is processed and represented in the brain. Classical studies using designed experiments have identified multiple dategory-sslectiin gions; however, these approaches rely on preconceived hypotheses about categories. Subsequent data-driven discovery methods have sought to address this limitation but are often limited by simple, typically linear encoding models. We propose an in silico approach for data-driven discovery of novel category-selectivig hypotheses based on an -ecoder transformer model. The architecture incorporates a prain-region to image-feature cross-attention mechanism, enabling nonlinear mappings between high- nigh-dimensional deep network features and semantic patterns encoded in the brain activity. We further introduce method to characterize the selectivity of individual parcels by leveraging diffusion based image generative models and large-scale datasets to synthesize and select images that maximally activate each parcel. Our approach reveals regions with complex, compositional selectivity involving diverse semantic concepts, which we validate in silico both within and across subjects. Using brain encoder "digital twin" offers pow- erful, data-driven framework for generating and testing hypotheses about visual selectivity in the human brain-hypothesess that guide future fMRI experiments. Our code is available at: https://Aricgecskotte- github.io/in silin--siio--ai maaicon/
1
Introduction
Over the past few decades, researchers have extensively studied the visual hierarchy in the brain, from early cortical areas that encode low-level features to higher-level regions that represent categorical information. Neuroimaging experiments, especially studies using functional magnetic resonance imaging (IMRI) have revealed specialized cortical regions for faces, places, words, bodies, and food 48
69 54]. However, visual perception is more nuanced than this short list of categories. remains unknown what additional visual concepts have
35 5
63
15 8 21
20
61 65
38
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

dedicated regions that enable humans to make sense of the complex world. Common mapping methods depend on cetpmmmenter-uuratee concepts, and empirically- data and expensive fMRI experiments. For example, one could present subject with large set of images and then label the selectivity of each cortical parcel by the images that elicit the strongest mean parcel response. As the stimulus set grows, however, the cost of data acquisition (operating the scanner, paying subjects, bonuses)-at least with current fMRI technology and experimental paradigms- may prove prohibitive. We propose methods that rely on the stale-of-the-ar encoding models to generalize to concepts beyond the stimuli for which fMRI responses have been measured.
alternatives require more
(a)
Brain
(b)
Parcel
responses
vertices
Image
Linear
Linear
encoder
Feedforward
Feedforward
Attention scores
Self Attention
Cross Attention
Scaled dot product
x N
Image patches (c)
Positional encoding
Parcel queries
Image tokens
Positional encoding
Parcel query
Subject specific
(d)
Shared across subjects
Skateboarding parcel Child eating parcel
Dynamic body parcel
Tool parcel
Figure/ 1: Brain encoder architecture and images predicted to maximally activate selected parcels outside the visual cortex. (a) Brain encoder architecture. (b) Cross-attention for parcel (MRI prediction. (c) A brain encoder ranks ImageNet [17] images by how much each image would activate a parcel. Images from two sample parcels, four images from a single subject are shown. (d) Images from two sample parcels, one image from each of the four subjects is shown for each parcel. All images are curated from the top 25, see Appendix A.7 for the parcel locations and full collage.
Our encoding model leverages recent advances AI and large-scale neural datasets to serve as "digital twin," upon which we perform extensive in silico experimentation 277 29 to generate hypotheses of complex categorical selectivity beyond the visual cortex. The encoding model enables us predict the neural activity of image sets far larger and more diverse than the limited number of images shown to the participant during IMRI, effectively expanding the search space for optimal visual stimuli. In addition, the encoding model is fully observable and differentiable, enabling interpretability queries using attention maps and gradient-based analyses, such as liffusion-basse models that gradients to find stimuli that elicit high activity [46].
Through experimentation with large image datasets on our model, we generated hypotheses for the selectivity of many parcels outside the visual area, well as optimal sets of images that maximally activate the parcels. These hypotheses can tested in targeted future fMRI experiments by showing only the sets of optimal stimuli, accelerating data collection and experiment iteration, and lowering

---

the cost of data acquisition. We demonstrate that our pipeline using brain encoders can test concepts that were explicitly shown to the subject in the scanner, effectively enriching the diversity and size of the fMRI training set.
Key contributions of our work:
1. Massive scale: applying in silico mapping on millions of images (ImageNet, BrainDIVE) with a wansformer-basee brain encoder, enabling discovery of parcel selectivity for concepts never shown in training. To the best of our knowledge, no other study has been done on this scale.
2. Mapping of the whole brain: expanding beyond visual cortex and revealing human-specific semantic selectivity.
3. In silico verification: our pipeline verifies selectivity hypotheses in silico with rigorous tests that evaluate how well label predict ground-truth activation on held-out set within and across subjects.
4. New IMRI experimental paradigm: as datasets grow and encoding models improve, our pipeline offers a way to leverage these advances to accelerate and improve the accuracy of whole-brain mapping.
2 Related Work
Semantic mapping. Our work builds upon growing body of computational modeling and machine learning research that investigates how semantic information represented in the higher visual cortex
[4]. Some approaches leverage large image datasets to build decoders [19 [2] 66 36 62 74
[4 44)
or models for generating optimal stimuli 13
46
45
28
58
47], while others use cross-domain (e.g.
30 These studies face key challenge of Our work seeks address this constraint by using encoders trained on large-scale datasets to perform in silico mapping. This allows us to expand the set of concepts that can be probed, beyond those stimuli shown to the subjects.
vision-language) mapping 33 32 37
50
dataset size, since the collection of neural data is often expensive and
Brain encoding models. Highlighting its importance, several community-drine efforts have sought to benchmark models predicting brain responses (60 68 26] With the release of increasingly large neural datasets each year, researchers have introduced novel architectures and methodologies to improve the accuracy of brain encoding model, including leveraging multiple datasets and pretrained networks [2] 43 773 III 72. 57 24 41 (7. While our paper uses a cutting-edge encoding model [3). our pipeline ultimately encoder-agnostic, and can use any encoder that is image- compuccom. As researchers build better brain encoders, we expect that the space of hypotheses our pipeline could generate and their accuracy will only grow.
Brain-optimized stimuli. Previous studies have introduced encoding model-based stimulus selec- tion and empirically validated the superstimuli in non-human primates and mice [10] 71 56 23 In
this work, we extend this general approach stimulus optimization for studying neural populations to the fMRI domain, revealing high-level human-specife selectivity beyond the visual cortex.
3 Methods
Our goal is to map the visual selectivity of parcels across the whole brain. First, train a brain encoder to predict IMRI responses from natural scene images. Then we select visually responsive and robust parcels for experimentation to determine categorical selectivity.
3.1 Parcellation Strategy
We partitioned the 327,684 cortical vertices across the whole brain into 1,000 functional parcels using the Schaefer resting-state functional connectivity parcellation (see Figure 2 (59].

---

Figure 2: Schaefer-1000 parcellation
3.2 Brain Encoder Architecture
Extending the work of Adeli et al. (2). our brain encoder predicts vertex-wise fMRI activity across the whole brain from input image (see Figure An image encoder backbone (see Section 4.2) first extracts patch embeddings from the image. A transformer decoder uses parcel-specific, learnable queries to attend to relevant patch embeddings via cross-attention. The decoder consists of a single transformer layer with cross-attention followed by a feedforward projection (see Figure L for cross- attention). Each output decoder token is linearly mapped to predict the fMRI responses for vertices in the corresponding parcel, and predictions from all output tokens are aggregated to obtain whole- brain prediction. Parcel queries, the transformer decoder, and linear mappings are optimized using Adam (39] to minimize the mean squared error between the predicted and actual fMRI responses. All other layers, including the backbone, are frozen. Separate models are trained for each subject.
To improve prediction accuracy, we ensemble multiple instances of the brain encoder. For each subject, we trained two random seeds with features from four different DINOv2 backbone layers (the Oth, 2nd, 4th, and 6th layers from the last). To predict a vertex, we take the weighted average across model predictions, scaled by softmax weights from validation set accuracy for each model on that vertex.
3.3 Parcel selection process for further experimentation
Not all regions in the brain are visually responsive, so we selected parcels for further experimentation
that satisfy three criteria: (1) Location: Fewer than 10% of parcel vertices overlap with the labeled area, since we are most interested in parcels beyond the visual cortex. (2) Visual responsiveness: The average noise ceiling must be in the top 25% of parcels that satisfy (1). (3) Model prediction accuracy:
The average prediction accuracy must be in the top 25% of parcels that satisfy (1).
Since the data quality varies slightly across subjects, we used percentiles rather than numerical cutoffs. The selection process for parcels in subject I's left hemisphere is shown in Figure ST in Appendix A.3
Cutoffs are determined separately for each hemisphere to maintain comparable parcel counts.
Among the 500 Schaefer parcels in each hemisphere for every subject, 409 parcels satisfy condition (1). After filtering out parcels with low mean noise ceiling or low mean model prediction accuracy, 179 + 12 parcels per subject are chosen for further experimentation.
3.4 Superstimulus Generation Process
We choose images that maximally activate (mean z-scored beta values) a parcel of interest using three different methods:
1. Natural Scenes Dataset (NSD) Ground Truth Images: Images from the held-out NSD test split, ranked based on ground truth data.
2. Diffusion--eneraate superstimuli: BrainDIVE 46 uses a generative backbone guided by gradients from brain encoder to generate images that can maximally activate specified brain parcels. We generated 400 images per parcel and reranked them with the brain encoder.
3. Encoder-selected ImageNet superstimuli: ImageNet the parcel, according to the encoder.
17 images that maximally activate

---

Experiments
4.1 Setup
We used the NSD [4]. the largest IMRI dataset to date, with 7T IMRI responses from 8 subjects who each viewed up 10,000 distinct natural scenes. Each image is presented up to three times, and our model is trained on the neural response averaged over the presentations. We report results for subjects who completed all NSD scan sessions (1, 2, 5, and 7), though we observed comparable results in all subjects. fMRI responses were preprocessed according to [4]. The resulting beta estimates were centered zero mean and scaled to unit variance before training and experiments. ROI labels were obtained from NSD. VI-hV4 ROIs are derived from a pRF experiment; body-, face-, place-, word-selective ROIs are derived from a fLoc experiment
4.2 Brain encoder results
Model prediction accuracy. Figure 3 shows the performance of the ensemble model with DINOv2 (ViT-B) backbone and the transformer cross-attentie mapping function for subject projected onto the cortical surface using Pycortex [25]. As expected, the model performs well on predicting the activity in the visual cortex (the area in the center of the flatmap), but also on several regions beyond the typical visual pathways.
Figure 3: Brain encoder prediction accuracy. Pearson correlation between model predictions and ground truth data for subject on the held-out test set.
Model comparison. We evaluate several encoding models with different backbones and mapping functions to find the most suitable model for whole-brain voxel-wise prediction. The Algonauts 2023 challenge [26] leaderboard showed that transformer-based backbones generally outperformed other model families such as convolutional networks (CNNs) at predicting fMRI activity. Therefore, we focused on evaluating features from several transformer backbones paired with either transformer attention-based mapping to neural data 2 or parameter-matcher model that linearly maps the CLS token to vertex values. For each architecture, we used an ensemble 16 models for each subject (4 feature backbone layers x hemispheres 2 random seeds). We compared the encoding accuracy, which is Pearson's 's correlation on the held-out set corrected noise ceilings (see Methods, Noise ceiling estimation).
Table 1: Brain encoder encoding accuracy using different architectures
Architecture
S1
S2
S5
S7
Backbone Mapping fn size (M)
DINOv2 (ViT-B) Linear
0.33
0.34
0.39
0.33
87 252
CLIP vision + Transf.
0.44
0,41
0,46
0,43
304 + 258
RADIOv2.5-H Transf.
0.26
0.35
0.34
0.30
652 + 258
DINOv2 (ViT-G w/reg)
Transf.
0.45
0.43
0.48
0.45
1136 + 258
DINOv2 (VIT-B) Transf.
0.45
0.43
0.48
0.43
87 + 258
Across the subjects that completed all NSD sean sessions (Table []. the transformer models with the DINOv2 backbone (53) outperform the other non-DINOv2 backbones. The two DINOv2 backbones perform similarly, which consistent with past work [ that showed diminishing or worse performance for DINOv2 models larger than the base (ViT-B). The transformer-base encoder
5

---

significantly outperformed the linear baseline, leveraging the attention mechanism to flexibly route information 6 [56] Figure 4 plots the difference prediction accuracy (Pearson 's correlation) between the two.
Figure 4 plots the difference in prediction aceuracy between DINOv2 and CLIP vision. While DINOv2 performs far better in the visual cortex, they perform similarly in many regions outside the visual area. For the rest of the paper, we performed our experiments the brain encoder with DINOv2 (ViT-B) + Transformer, since it offered the best overall speed-performance tradeoff.
(a)
(b)
Plerson Mad
Figure 4: (a) DINOv2 Transformer Linear head-to-head comparison. Difference in prediction accuracy for subject between DINOv2 (ViT-B) with transformer vs. linear mapping functions (> or red means transformer is better). (b) DINOv2 CLIP head-to-head comparison. Difference in prediction accuracy for subject between DINOv2 (VīT-B) and CLIP vision (> 0 or red means DINOv2 is better).
4.3
Sanity check known regions
We first validate our paradigm by replicating the well-documented category selectivity of ventral pathway categorical areas. We show the results for sample parcel from the labeled area, one which significantly overlaps with afLee (47.6% of the vertices in the parcel overlap with aTL--face). As shown in Figure [5] the images maximally activating the parcel overlapping with aTL-faces prominently feature faces, which agrees with previous work on the selectivity of this area [63] In Appendix A.2 we reproduce the selectivity of body-, place-, and word-selective areas. Appendix A.4 we show that stimuli generated by BrainDIVE. reproduce the fine-to-coarse visual hierarchy progressing from V1-V4 to FFA.
4.4 Extension to unlabeled areas
We now extend our analyses to parcels beyond the visual cortex. We show images selected by the encoder to maximally activate each parcel for qualitative evaluation, then verify that our labels indeed explain parcel activations using formal statistical tests.
Labeling unlabeled parcels. We apply the same image-selection method to generate selectivity hypotheses for unlabeled parcels. Figure displays ImageNet images predicted by the encoder to maximally activate two unlabeled subject-specifie parcels: one appears to depict skateboarding, the other a child eating food. Full results, including parcel location and the corresponding NSD and BrainDIVE images,
shown in Figures [S3 and [S6 (Appendix) (.7). Many parcels with selectivity that appeared to be consistent and complex were found across the whole brain, suggesting that there exist regions in the brain that respond more complex concepts than the basic categories labeled in visual cortex.
Evaluating our labels. Our sanity checks demonstrated that our brain encoder retrieves images that align with the category selectivity of the labeled parcels. However, since the parcels we interested in are outside the visual area and therefore unlabeled, we would like to quantitatively show that the images selected by encoder faithfully reflect underlying neural selectivity.
conducted two tests against the NSD test set-the only held-out fMRT data not used during training. If the selected images (hereafter "labels") genuinely reflect parcel selectivity, other images with greater semantic similarity to the label should result in higher parcel activation. Prior studies have shown that categorical selectivity is not binary but graded across diverse stimuli [2] 49 For

---

(a)
B
(e)
Imagenel vs Gen. Images Activation Dist
70000
Imagenet
Gen
60000
Imagenet CI Lower
50000
Imagenet Upper
Lower
L
R
30000
20000
10000
F
Avsnage cledi activalion magritude (escoore))
(b)
NSD
(c)
Generated
(d)
Imagenet
Figure 5: Verifying the selectivity of aTLL-faces. (a) The location of the parcel. The inflated cortical surface (ventral view), with the left hemisphere on the left. (b) Held-out NSD images that maximally activate the parcel (based on ground-true -ruth fMRI). (c) BrainDIVE generated and re-ranked images optimized for the parcel. (d) (e) Distribution of predicted parcel activation for all of ImageNet images compared to images in (c).
images from ImageNet according to the encoder.
example, fMRI responses to 1,705 object and action categories revealed that brain-response similarity strongly correlates with semantic similarity [33]. Accordingly, we expect parcel's ' mean activity to scale with how semantically close image to that parcel's 's preferred concept, defined by encoder's maximally activating examples.
In each test, we compared two predictors: (1) labels derived from our encoder, and (2) a baseline that forms its hypothesis from each parcel's most activating NSD-training images, selected using their measured responses. This baseline mirrors conventional fMRI studies that rely solely on stimuli shown in the scanner. our encoder performs better than the baseline training set, shows we can use brain encoders to discover novel concepts not explicitly shown to the subject.
asked whether semantic similarity NSD test images better than chance. Our pipeline represents a label as the mean CLIP embedding of the top 32 ImageNet BrainDIVE images (both shown separately); the baseline uses the top 32 NSD-training images. Appendix for which each label significantly outperforms random ranking. To ensure that the results reflect an expansion of the stimulus space and not merely dense sampling near peak activation, Appendix I6 reports the same results with varying number of top images used to create the label. Each parcel whose model-derived label outperforms the null is deemed successfully labeled. Across tests, the encoder-selected stimuli generally outperformed the baseline, labeling greater number of parcels.
For Test
label predicted the activation rank order of
details the procedure. Table Tablepreports the number of parcels
Table 2: Fraction of parcels whose model- label predicts parcel activation rankings significantly better than chance (p 0.05, FDR corrected)
Model type
S1
S2
S5
S7
NSD train
150
/181
163/192
136/175
155/1966 150/196
Our encoder w/ImageNet
139
1811
167/192
130/
Our encoder w/BrainDIVE
135/ /181
170/192
139 339/175
156/196
7

---

For Test 2, we compare Spearman's 's rank correlations between the ground-truti activation ordering and each model's predicted ordering, quantifying which model's selectivity hypothesis best explains parcel activations. Table 3 summarizes these coefficients across all parcels. A head-to-head comparison between our encoder and the baseline appears Table X (Appendix A.13)..
Table 3: Spearman's (mean + std) between the model-predicted and ground-truth activation rankings on the NSD test set, averaged across parcels.
Model type
$1
S2
S5
S7
Null
0.000 = 0.045
0.000
0.000
0.000
NSD train
0.162 0.098
0.164 + 0.091
0.150 + 0.094
0.148 + 0.086
Our encoder w/NNet
0.168 0.106
0.163 0.082
0.142 0.092
0.133 0.075
Our encoder w/BD
0.163 = 0.121
0.190 - 0.099
0.154 + 0.094
0.133 + 0.083
Notes. INet ImageNet.
BD BrainDIVE.
The rank ordering derived from the encoder-selected stimuli generally outperforms the baseline ordering based on the NSD test set. To contextualize the rank correlation magnitude, we report in (A.endixXA.1jtthie results for known parcels in the visual area as a benchmark for what constitutes meaningful selectivity. The average rank correlation is in the range of areas like PPA and RSC-both widely studied and accepted in the literature. Because these encoder-driven selectivity hypotheses better explain activation patterns the NSD test images, our results indicate that the brain-encoder pipeline can generate finer-grained categorical hypotheses than those afforded by the stimuli actually shown to the subject in the scanner.
Choosing parcels for future fMRI experimentation. For an fMRI study aimed uncovering the selectivity of parcels outside visual cortex, an experimenter may be interested in parcels whose activity can be well explained by a semantic label. To prioritize such parcels, we define a metric that ranks the quality of the hypotheses generated by our pipeline. In Figure 6 we plot the proportion of top 32 maximally- acttivatin images successfully retrieved against the number of images retrieved by the pipeline. The top five parcels per subject (ranked by area under the curve) are highlighted.
NSD test Imagenet Retrieval Rale
SeGNT2
10-
STACe
85%
SELL
Number factiend Imapa
Runber di Earwwed Images
Numbar of Fast-oed Itages
Hurber of Datroved Imagus
Figure 6: Retrieval accuracy of parcel labels. Using the concept vector the label for each parcel, we retrieve a varying number of images from the test set based on cosine similarity, and calculate the fraction of overlap with 32 maximally acttivatnng NSD test images (i.e. recall, the y-axis). Curves that rise quickly and plateau high indicate concept vectors whose semantics closely match the parcel's true selectivity.
From an experimenter's perspective, the parcels shown in Figure 6 may be promising targets for follow-up fMRI studies, since semantic labels seem to capture the parcel selectivity exceptionally well. Our pipeline identifies images whose semantic representations appear maximally activate these parcels- at least insofar as this can be evaluated with the modest NSD test set.
4.5 Mapping Cross-subject Selectivity
We now explore the selectivity of a parcel outside visual cortex (Figure 7A) by examining ImageNet images that the encoder predicts will maximally activate it. From the top 9 images per subject, we display 3 from each participant who completed all NSD scanning sessions. These maximally- activating images lie substantially outside the parcel's activation distribution for the remainder of
8

---

ImageNet (Figure 7p).
The selected images consistently depict hands manipulating tools-such as writing instruments or cooking utensils (see full collages in Appendix (A99). When asked identify common theme among the top 25 ImageNet images, ChatGPT likewise highlights hands with objects. Recent work has reported tool-use representations in nearby cortical regions [16]
(a)
(b) S1
...utensiis in action, often practical, hands-on settings..."
"...hands actively engaging in step of making food..."
S2
"...objects body parts being used writing instruments tools...
(c)
S5
Inagenet
Alleageet
80008
Alteagenn tower
Allvigerel
20000
Iragire an Cilluee
"...action involving tools sports equipment..."
$2.00
S7
Figure 7: (a) Parcel location. (b) Selected from the images in ImageNet that maximally activate parcel of interest, across subjects I, 5, and 7. ChatGPT 4o generated labels for top-25 ImageNet images, shown on right (see Appendix AA.18 for details). (c) A comparison of the activation magnitude of the parcel from all ImageNet images and the top 9 images, with 95% confidence intervals.
Evaluating our labels. We qualitatively showed that our pipeline can uncover a parcel exhibiting consistent semantic selectivity across subjects. We now seek to quantitatively verify these hypotheses on ground-truth data. We ran statistical test similar to that in Section 4.4 with key differences because we are evaluating selectivity across subjects.
1. Combining across subjects: For each parcel, we form the optimal image set by combining the top-32 maximally-activalae images chosen by encoder models trained on the other three subjects.
2. Retrieval set: Because each subject's NSD training set contains distinct images, the ranking retrieval is performed on that subject's own NSD training set.
We again compare the selectivity hypotheses generated by our brain-encoded pipeline with the NSD- training-set baseline (described in the within-subject statistical tests). Table Table/Irpport, reports the results of this modified test (cf. Table 2); additional details are provided in Appendix A.11
Table 4: Fraction of parcels shared across subjects whose model-derived label predicts parcel activation rankings significantly better than chance (p 0.05, FDR corrected)
Model type
S1
S2
S5
S7
NSD train
97/49
44/49
40/49
40/49
Our encoder w/ImageNet
38 38/49
45/49
42 42/49
39/49
Our encoder w/BrainDIVE
37/49
40/49
38/49
38/49

---

Comparing the number of parcels for which each hypothesized label ranks NSD-training images better than chance, our pipeline generally outperforms the NSD-train baseline. These findings indicate that for parcels with high visual responsiveness shared across subjects, our pipeline can generate better hypotheses that describe the shared selectivity.
presents results from the same statistical test used in Table 3 reporting the average Spearman rank correlations for each model across parcels. A head-to-head comparison appears in Table (Appendix A. 1 and Figure (aaeeeSSsppppp (Appendix A. 14) highlights the most promising cross-subject parcels for future tMRI experiments.
presents
Table 5: Spearman's / p (mean + std) between the model-predicted (from all subjects other than heldout) and ground-truth activation rankings the NSD training set, averaged across parcels.
Model type
S1
S2
SS
S7
Null
0.000=0.00 0.011
0.000
0.000:10.011
0.000-10.011 0.091 + 0.070
NSD train
0.068 = 0.068
0.094 + 0.065
0.090 + 0.074
Our encoder w/lNet
0.072 0.071
0.105 = 0.073
0.105 + 0.078
0.093 = 0.075
Our encoder w/BD
0.078 + 0.077
0.103 0.087
0.106 0.091
0.089 0.079
Notes. INct = ImageNet. BD = BrainDIVE.
5
Discussion
Leveraging recent advances in Al and the availability of large-scale datasets, we introduce a data- driven paradigm for discovering parcel selectivity beyond visual cortex, paving the way toward systematic whole-brain labeling of higher-order visual representations. In particular, our transformer- based encoder, with its cross-attention mechanism and nonlinear mappings, routes visual information more effectively for parcels in and outside classical visual areas.
Limitations. Although we predict images that should maximally activate a parcel, have not yet validated these "superstimuli" in new fMRI experiments. On a held-out NSD split we show that our semantic labels predict parcel activations well, but whether the synthesized superstimuli elicit even stronger responses remains unknown. Moreover, because the encoder was trained solely on NSD may inherit dataset biases. For example, one parcel in subject 2's left hemisphere appears zebra-selective (Figure SSO Appendix 19.19.. however, this could be due to the overeeepresnnttation of giraffe and zebra images the dataset. We did not quantify how such biases affect our results; future work should train on multiple datasets to address this concern.
More fundamentally, category selectivity is limited theoretical construct for understanding brain computation. We hope our approach will help the field go beyond the technical limitations of prior work and help reveal the fundamental limits of understanding brain computation through the lens of category selectivity.
Optimizing future fMRI experiments. Our framework paves the way for future fMRI studies that mitigate the effect of small datasets and experimenter bias in image selection by demonstrating the promise of in silico mapping to superstimuli. These superstimuli can then be tested in follow-up scans to verify the selectivity of newly discovered areas. By letting the encoder drive hypothesis generation, researchers can discover optimal stimuli empirically even for concepts never presented in the scanner maximizing data- collectoooetiinnnc efficiency. Because our encoder architecture is modality- agnostic, the same semantic-nn maaantic-maaping approach could be extended multi-modal backbones, enabling superstimuli generation across sensory domains.
10

---

Acknowledgments
Research reported in this publication was supported in part by the National Institute of Neurological Disorders and Stroke of the National Institutes of Health under award numbers IRFINSI LINS1288 RKFINS128897 and 1801NS128897. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
References
[1] Hossein Adeli, Seoyoung Ahn, Nikolaus Kriegeskorte, and Gregory Zelinsky. Affinity-based attention self-supervised transformers predicts dynamics of object grouping humans, 2023.
httttttts:///rxiiv wwwp:/wwwwwv.ra/////aaaaaa 00294
[2] Hossein Adeli, Sun Minni, and Nikolaus Kriegeskorte, Predicting brain activity using Trans- formers, August 2023. URL http:p///bbooztrrr./ooooo/////// 551743
[3] Hossein Adeli, Minni Sun, and Nikolaus Kriegeskorte. Transformer brain encoders explain human high-level visual responses, May 2025. URL http:://///axi wwwp:iii.rrg//baa htte//bb/2500.... arXiv:2505.11 17329 [q-bio].
http:p///arii.oorrrabb////.0... 17329
[4] Emily J. Allen, Ghislain St- Yve,, Yihan Wu, Jesse L. Breedlove, Jacob S. Prince, Logan T. Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, Benjamin Hutchin- son, Thomas Naselaris, and Kendrick Kay. A massive 7T fMRI dataset to bridge cog- nitive neuroscience and artificial intelligence. Nature Neuroscience, 25 Jan- doi: 10.1009441193-001-009662
uary 2022. ISSN 1097-6256. 1546-1726. httpps:/vit //www http//w...auur...emmm nature. htt.a/rr.cccorrrrcc//////11233 -20-0000-0----22-2
URL
[5] Truett Allison, Gregory McCarthy, Anna Nobre, Aina Puce, and Aysenil Belger. Human extrastriate visual cortex and the perception of faces, words, numbers, and colors. Cerebral cortex, 4(5):544 554, 1994.
[6] Mehdi Azabou, Vinam Arora, Venkataramana Ganesh, Ximeng Mao, Santosh Nachimuthu, Michael Mendelson, Blake Richards, Matthew Perich, Guillaume Lajoie, and Eva L. Dyer. A unified, scalable framework for neural population decoding. Thirty-seventh Conference on Neural Information Processing Systems, 2023.
[7] Mehdi Azabou, Vinam Arora, Venkataramana Ganesh, Ximeng Mao, Santosh Nachimuthu, Michael J. Mendelson, Blake Richards, Matthew G. Perich, Guillaume Lajoie, and Eva Dyer. A unified, scalable framework for neural population decoding, 2023. URL https
//arxiv.. wwwooogggbbbo httg/abs/231..10000
[8] Chris I. Baker, Jia Liu, Lawrence Wald, Kenneth K. Kwong, Thomas Benner, and Nancy Kanwisher. Visual word processing and experiential origins of functional selectivity in human extrastriate cortex. Proceedings of the National Academy of Sciences of the United States of America,
May 2007. ISSN 0027-8424
doi: 10.1073/pnas.
0703300104.
Place: United States.
[9] Alfonso Barrós- Loscerallsse,,,ess. Julio González, Friedemann Pulvermüller, Noelia Ventura-Campont Juan Carlos Bustamante, Víctor Costumero, María Antonia Parcet, and César Ávila. Reading salt activates gustatory brain regions: IMRI evidence for semantic grounding in novel sensory modality. Cerebral cortex (New York, N.Y. : 1991), 22(11):2554-2563 November 2012. ISSN doi: 10. 00.1993/cercor//hh224,, Place: United States.
1460-2199 1047-3211.
[10] Pouya Bashivan, Kohitij Kar, and James J DiCarlo. Neural population control via deep image synthesis. Science, 36+(6439):aavv443, 2019.
[11] Roman Beliy, Navve Wasserman, Amit Zalcher, and Michal Irani. The wisdom crowd of brains: A universal brain encoder, 2025. htthtt // /arxivi. org/ horg/abs/44000 12179

---

[12] Edwin J. Burns, Taylor Amold, and Cindy M. Bukach. P-curving the fusiform face area: Meta- Meta-analyses support the expertise hypothesis. Neuroscience & Biobehavioral Reviews, 104: 209-221, 2019. ISSN 0149-7634. doi: https///aaoooooooolll////eeeeeee
URI.
htttps://ww.. http:://www...eaeee.......aarrra
[13] Diego García Cerdas, Christina Sartzetaki, Magnus Petersen, Gemma Roig. Pascal Mettes, and Iris Groen. Brainactiv: Identifying visuo-semantic properties driving cortical selectivity using diffusion-based image manipulation. bioRxiv, 2024. doi: 10.1 1011102/20 10../200............. 29.09989........ URL httpss//www.
www.:biorxv..ooo org http://ttteteee/l///////////////
10. 29. 29.29962088
[14] Zijiao Chen, Jiaxin Qing, Tiange Xiang, Wan Lin Yue, and Juan Helen Zhou. Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding. arXiv preprint wwXivv2211.06665 .6955, 1(2)4, 2022.
[15] I.. Cohen, S. Dehaene, L., Naccache, S. Lehéricy, G. Dehaene-Lambertz M. A. Hénaff, and F. Michel. The visual word form area: spatial and temporal characterization of an initial stage of reading in normal subjects and posterior split-brain patients. Brain a journal of neurology, 123 ( (t 2):291-307, February 2000. ISSN 0006-8950. doi: 10. (0.1009/raain/123.2222 Place: England.
[16] Davide Cortinovis, Marius V. Peelen, and Stefania Bracci. Tool Representations Human Visual Cortex. Journal of Cognitive Neuroscience, 37(3): 531, March 2025. ISSN 0898- 929X. doi: 10. 1621/o/eoonn__ a 02281. URL_https://dol /rgp::////o/ooo///////////////// 10t. 1162/jocn..nno a_ a_02281
[17] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A large- scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255. Miami, FL. June 2009. IEEE. ISBN 978- 42444-999-88 doi: 10. 2BW.CCCVR..00.55266882 URL http:://eeeeeepplrrr ieee..rrree.eeeee...e....eee.eee www.eeeerg//eee owo http//dccmmmnt//22666442/
[18] R. Desimone and J. Schein. Visual properties of neurons in area V4 of the macaque: sensitivity to stimulus form. Journal of neurophysiology, 57(3):835-86,, March 1987. ISSN 0022-3077. doi: 10.1 10.1152/jn. 09887.7.3388.
Place: United States.
[19] Adrien Doerig, Tim C Kietzmann, Emily Allen, Yihan Wu, Thomas Naselaris, Kendrick Kay, and Ian Charest. Semantic scene descriptions as an objective of human vision. arXiv preprint arXiv:2209. 1137,, 2022.
[20] Paul E Downing, Yuhong Jiang, Miles Shuman, and Nancy Kanwisher, A cortical area selective for visual processing of the human body. Science, 293(5539):2470-4444 2001.
[21] Russell Epseiin and Nancy Kanwisher. A cortical representation of the local visual environment. Nature, 392(6676))598-60. 1998.
[22] Matteo Ferrante, Furkan Ozcelik, Tommaso Boccato, Rufin VanRullen, and Nicola Toschi. Brain captioning: Decoding human brain activity into images and text. arXiv preprint arXiv:2305. 2023.
[23] Katrin Franke, Nikos Karantzas, Konstantin Willeke, Maria Diamantaki, Kandan Ramakrishnan, Pavithra Elumalai, Kelli Restivo, Paul Fahey, Cate Nealley, Tori Shinn, et al. Dual-feature selectivity enables bidirectional coding in visual cortical neurons. bioRxiv, pages 2025 2025.
[24] Maëlle Freteault, Maximilien Le Clei, Loic Tetrel, Lune Bellec, and Nicolas Farrugia. Alignment of auditory artificial networks with massive individual fmri brain data leads to generalisable improvements in brain encoding and downstream tasks. Imaging Neuroscience, 3. 04 2025. ISSN 2837-6056. doi: 10.1162/imag/ al 00525. htttt:://///d /ttp::o///oooor//11... http:/10..11///11 1162/imag. a_00525
[25] James S. Gao, Alexander G. Huth, Mark D. Lescroart, and Jack L. Gallant. Pycortex: an interactive surface visualizer for fMRI. Frontiers in Neuroinformaticc, 9, September 2015. ISSN 1662-5196. doi: 10.33899hin222255500002. URL. http://journal. frontiersin
org/ heg/Article/ie 10. 3389/ fninf. 2015. 00023/00020000 abstract
12

---

[26] Alessandro T Gifford, Benjamin Lahner, Sari Saba-Sadiya, Martina G Vilas, Alex Las- celles, Aude Oliva, Kendrick Kay, Gemma Roig, and Radoslaw M Cichy. The algonauts project 2023 challenge: How the human brain makes sense of natural scenes. arXiv preprint arXiv:2300.00 0033000..099,.... 2023.
[27] Alessandro T. Gifford, Maya A. Jastrzębowska, Johannes J. D. Singer, and Radoslaw M. Cichy. silico discovery of representational relationships across visual cortex, 2025. URL attps://arxie org/ abs /411 10872
[28] Zijin Gu, Keith Wakefield Jamison, Meenakshi Khosla, Emily J. Allen, Yihan Wu, Ghislain St-Yves, Thomas Naselaris, Kendrick Kay, Mert R. Sabuncu, and Amy Kuceyeski. NeuroGen: Activation optimized image synthesis for discovery neuroscience. Neurolmage, 247:118812 February 2022. ISSN 10538119, 19. doi: 10.10166/ http::/o.eeurrmmmae00011111111.1
URL https
//linkinghub. streghub.eseevierr.. elsevier com cttp://eerrreeeep////11111111110
[29] Umut Güçlü and Marcel A. van Gerven. Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. Journal of Neuroscience, 35 127:100055-0014, 2015. ISSN 0270-6474, doi: 10. 152//NEUROSSL15022
URL
https://www. https://ww.jjeuuossci onc.rosii. httg//ottent//////22 10005
[30] Tomoyasu Horikawa, Alan S. Cowen, Dacher Keltner, and Yukiyasu Kamitani. The Neural Representation of Visually Evoked Emotion Is High- bingh-Dimessioall. Categorical, and Distributed across Transmodal Brain Regions. iScience, 23(5), May 2020. ISSN 2589-0042. doi: 10. 1016/ j.isci.2020.1 101060. URL https: org/ 10. 1016/ isci 2020 101060 Elsevier.
Publisher:
[31] D. H. Hubel and T. N. Wiesel. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. The Journal of physiology, 160(1):106-154, January 1962. ISSN 0022-3751 1469-7793. doi: 10.1 1ttp:phssooo.192.p0000....
Place: England.
[32] Alexander G. Huth, Wendy A. De Heer, Thomas L. Griffiths, Frédéric E. Theunissen, and Jack L. Gallant. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature, 532 (7600):453-458, April 2016. ISSN 0028-0836. 1476-4687. doi: 10.1038/matrre17633 URL
htttps://wat //wwww http:///wwwwatuueeeetmme nature..... wtt.//attieeee//ntttreeeeeeeeeee
[33] Alexander G. Huth. Shinji Nishimoto, An T. Vu, and Jack L. Gallant. A Continuous Semantic Space Describes the Representation of Thousands of Object and Action Categories across the Human Brain. Neuron, 7666)1210--224, December 2012. ISSN 08966273. doi: 10. m0166/..euron..222...0 50896627312009349
URL htll.httss:/// /
elsevier.
[34] Nidhi Jain, Aria Wang, Margaret M. Henderson, Ruogu Lin, Jacob S. Prince, Michael J. Tarr, and Leila Wehbe. Selectivity for food in human ventral visual cortex. Communications Biology 2023 6:1, 6:1-14, 2 2023. ISSN 2399-3642. doi: 10. 10088844200--023--4445---
[35] Nancy Kanwisher, Josh McDermott, and Marvin Chun. The fusiform face area: a module in human extrastriate cortex specialized for face perception. Journal of neuroscience, 17(11): 4302-4311, 1997.
[36] Kendrick N Kay, Thomas Naselaris, Ryan J Prenger, and Jack Gallant. Identifying natural images from human brain activity. Nature, 452(7185):352-33) 2008.
[37] Alexander J. E. Kell, Daniel L. K. Yamins, Erica N. Shook, Sam V. Norman-Haiinere, and Josh H. McDermott, A Task-Optimized Neural Network Replicates Human Auditory Behavior, Predicts Brain Responses, and Reveals a Cortical Processing Hierarchy. Neuron, 98(3):630 644.e16, May 2018. ISSN 1097-4199 1896-6273 United States.
doi: 10.1016//.neuron.200 wet.:eeooooo.1......... 8.03.044. Place:
[38] Meenakshi Khosla, N. Apurva Ratan Murty, and Nancy Kanwisher. A highly selective response to food human visual cortex revealed by hypothesis-free voxel decomposition. Current Biology, 12:1-13,20 2022.
13

---

[39] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[40] Ru Kong, Yan Rui Tan, Naren Wulan, Leon Qi Rong Ooi, Seyedeh-Rezvane Farahibozorg, Samuel Harrison, Janine D. Bijsterbosch, Boris C. Bernhardt, Simon Eickhoff, and B. T. Thomas Yeo. Comparison between gradients and parcellations for functional connectivity prediction of behavior. Neurolmage, 273:140044, June 2023. ISSN 1095-9572 1053-8119. doi: 2e0.0iof. 120044.
Place: United States.
[41] Nikolaus Kriegeskorte. Deep neural networks: a new framework for modeling biological vision and brain information processing. Annual review of vision science, 1:417-446, 2015.
[42] Nikolaus Kriegeskorte, Marieke Mur, Douglas A. Ruff, Roozbeh Kiani, Jerzy Bodurka, Hossein Esteky. Keiji Tanaka, and Peter A. Bandettini. Matching categorical object representations in inferior temporal cortex of man and monkey. Neuron, 60(6): Dec 2008. ISSN 0896- 10.043. URL https: www.:oo..ooog..oooooooo 10. 1016/j neuron
6273. doi: 1008.10.040 043
[43] Yuanning Li, Huzheng Yang, and Shi Gu. Upgrading voxel-wise encoding model via integrated integration over features and brain networks. bioRxiv, 2022. doi: 10. 101110022 56.551387 URL httpss:/www.w.... biorxiv org/ http://o...ooteee/aaaa/////////0 1ttp://2222......11....... .1507.
[44] Yulong Liu, Yongqiang Ma, Wei Zhou, Guibo Zhu, and Nanning Zheng. Brainclip: Bridging brain and visual-linguistic representation via clip for generic natural visual stimulus decoding from fmri. arXiv preprint arXiv:2302.12977 2023.
[45] Andrew Luo, Margaret Marie Henderson, Michael J. Tarr, and Leila Wehbe. Brainscuba: Fine-grained natural language captions of visual cortex selectivity. In The Twelfth International Conference on Learning Representations, 2024. URL
id=UUUHUUUTRU
[46] Andrew Luo, Margaret M. Henderson, Leila Wehbe, and Michael Tarr. Brain Diffusion for Visual Exploration: Cortical Discovery using Large Scale Generative Models, November 2023. 03089 arXiv:2306.0308 [cs].
URL httt:://arrat http://ariii.orrra////eae//
[47] Takuya Matsuyama, Shinji Nishimoto, and Yu Takagi. LaVCa: LLM-assisted Visual Cortex Captioning, February 2025. URL http: http:///aarddv..r/aas/////////// aww.g//abb///2/// 13606 13X6v:2502.13666 [q-bio].
[48] Gregory McCarthy, Aina Puce, John C Gore, and Truett Allison. Face-specific processing in the human fusiform gyrus. Journal of cognitive neuroscience, 9(5):005-61) 610, 1997.
[49] Marieke Mur, Douglas A. Ruff, Jerzy Bodurka, Peter De Weerd, Peter A. Bandettini, and Nikolaus Kriegeskorte. Categorical, yet graded single-image activation profiles of human category-selective cortical regions. Journal of Neuroscience, (2(25))86499 2012. ISSN 0270-6474. doi: 10. 111/////EEOOOII.23330 11.2012. content/32/25/8GA4
btcttps:/wwwwjjeneee
org/
[50] Till Nierhaus, Sara Wesolek, Daniel Pach, Claudia M. Witt, Felix Blankenburg, and Timo Schmidt. Content Representation of Tactile Mental Imagery in Primary Somatosensory Cortex. eNeuro, I0K6):ENEURO., CEUUROO008---
June 2023. ISSN 2373-2822 doi: 10.1523/
04408-2 2088-22.0023
Place: United States.
[51] Toshiki Okumura, Ikuhiro Kida, Atsushi Yokoi, Tomoya Nakai, Shinji Nishimoto, Kazushige Touhara, and Masako Okamoto. Semantic context-dependent neural representations of odors in the human piriform cortex revealed by 7T MRI. Human brain mapping, 45(6))26661, April 2024. ISSN 1097-0193 1065-9471.
doi: 10. 10.100//bb.226811
Place: United States.
[52] OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Flo- rencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett- Shaaaddr--SShprro. Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna- Brakman,
14

---

Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, An- drew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael. Brooke Chan, Che Chang. Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung. Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao. Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopss, Jonathan Gordon. Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, tukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Lukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Man- ning. Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O' Oeeefe Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pan- tuliano. Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng. Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto. Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry. Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Kata- rina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang. Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss. Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu. Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo. Kevin Yu, Qiming Yuan. Wojeiech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024. URL https: //wrxxavavvvv///////a//aaaawwav org/ org/abs//////eooeee/..ooee 2200./233...2 08774
[53] Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khali- dov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacldin El-Nouby, Mahmoud Ass- ran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang- Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning Robust Visual Features without Supervision, February 2024. http:/// orw.oogggaooo..oo 200ps//3344.77 07193 arXiv:2304.07193 [cs].
[54] Marius V. Peelen and Paul E. Downing. Selectivity for the human body in the fusiform gyrus. Journal of neurophysiology, 93(1):603-608, January 2005. ISSN 0022-3077. doi:
10.1152/jn.00551322000 Place: United States.
[55] Francisco Pereira, Bin Lou, Brianna Pritchett, Samuel Ritter, Samuel J. Gershman, Nancy Kanwisher, Matthew Botvinick, and Evelina Fedorenko. Toward a universal decoder of linguistic meaning from brain activation. Nature Communicattinns, 9(1):963, March 2018.
15

---

ISSN 2041-1723, doi: 10.1038//41447--0188000066 URL https://doi.on 841467-018-0366-4
[56] Pawel A Pierzchlewicz, Konstantin Willeke, Arne Nix, Pavithra Elumalai, Kelli Restivo, Tori Shinn, Cate Nealley, Gabrielle Rodriguez, Saumil Patel, Katrin Franke, et al. Energy guided diffusion for generating neurally exciting images. In Proceedings of the 37th International Conference Neural Information Processing Systems, pages 32574-32600 32601, 2023.
[57] Xuelin Qian, Yun Wang. Jingyang Huo, Xinwei Sun, Yanwei Fu, and Jianfeng Feng. fMRI-PTE: A large-scale fMRI pretrained transformer encoder for multi-subject brain activity decoding, 2024. URL htttts://ppppreeeeeeee
Computational models of http://wweeeeooeom/////////a/p/a selectivity. Nature communications, 12(1)) 2021.
[58] N Apurva Ratan Murty, Pouya Bashivan, Alex Abate, James DiCarlo, and Nancy Kanwisher. selective brain regions enable high-throughput tests of
[59] Alexander Schaefer, Ru Kong. Evan M Gordon, Timothy Laumann, Xi-Nian Zuo, Avram J Holmes, Simon B Eickhoff, and B T Thomas Yeo. Local-Global Parcellation of the Human Cerebral Cortex from Intrinsic Functional Connectivity MRI. Cerebral Cortex, 28(9):3095- 3114. September 2018. ISSN 1047-3211, 1460-2199, doi: 10. 10.109//cerror/bbxxI1 179. URL
https://academin oup wtt www.o///eeoo//aareeeeeeoo htw.rrcooraaticcee/////////// /0tp//////99/93988000
[60] Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J Majaj, Rishi Rajalingham, Elias B Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy Roy, Franziska Geiger, et al. Brain-score: Which artificial neural network for object recognition is most brain-like? BioRxiv, page 407007, 2018.
[61] Rebecca F. Schwarzlose, Chris Baker, and Nancy Kanwisher. Separate face and body selectivity on the fusiform gyrus. The Journal of neuroscience : the official journal of the Society for Neuroscience, 25(47) 11059, November 2005. ISSN 1529-2401 0270-6474. doi: 2011522/NEUUROS00222--555555
Place: United States.
[62] Paul S Scotti, Mihir Tripathy, Cesar Kadir Torrico Villanueva, Reese Kneeland, Tong Chen, Ashutosh Narang, Charan Santhirasegaran, Jonathan Xu, Thomas Naselaris, Kenneth A Norman, al. Mindeye2: Shared-subject models enable fmri-to-image with hour of data. arXiv preprint arXiv:2403.J 1207, 2024.
[63] Justine Sergent, Shinsuke Ohta, and Brenna Macdonald. Functional Neuroanatomy of Face and Object Processing: A Positron Emission Tomography Study. Brain, 1 115(1):15 36, February 1992. ISSN -950- doi: 10.1093/brain/115.... lttps:/// org/10. 1093/ brain/115. 1. 15 _eprint:
1-15.pdf.
[64] Irina Simanova, Peter Hagoort, Robert Oostenveld, and Marcel A. J. van Gerven. Modality- Independent Decoding of Semantic Information from the Human Brain. Cerebral Cortex, 24 (2):426 434, February 2014. ISSN 1047-3211. doi: 10. 10.1093/cerco//bhs3344. URL https 7/doi.org/f 1093/ cercor/bhs324
[65] W. Kyle Simmons, Alex Martin, and Lawrence W. Barsalou. Pictures of appetizing foods activate gustatory cortices for taste and reward. Cerebral cortex (New York, N.Y. 1997), 15(10): 1602-1608, October 2005, ISSN 1047-3211, doi: 10.1003/cercor//hh003.. Place: United States.
[66] Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. bioRxiv, pages 2022-11, 2022.
[67] Jerry Tang, Meng Du, Vy Vo. Vasudev Lal, and Alexander Huth. Brain encoding models based on multimodal transformers can transfer across language and vision. Advances in neural information processing systems, 36:29654-29666, 12 2023.
[68] Polina Turishcheva, Paul G Fahey, Michaela Vystrčilová, Laura Hansel, Rachel Froebe, Kayla Ponder, Yongrong Qiu, Konstantin F Willeke, Mohammad Bashiri, Eric Wang, et al. The dynamic sensorium competition for predicting large-scale mouse visual cortex activity from videos. ArXiv, pages arXiv-2305, 2024.
16

---

[69] L. N. van der Laan, D. T. D. de Ridder, M. A. Viergever, and P. A. M. Smeets. The first always with the eyes: meta-analysis on the neural correlates of processing visual food cues. Neurolmage, 25(1):29--333, March 2011. ISSN 1095-9572 1053-8119. doi: n..0066i.nermiimaee.000000
1.55..
Place: United States.
[70] A. van der Schaaf and J.H. van Hateren. Modelling the power spectra of natural images: Statistics and information. Vision Research, 36(17):2759-277 1996. ISSN 0042-6989. doi: https://doi.org///O 111.0010000000229990999990990990
8. URL
www.nccciiiee.r.tttt............ com/
[71] Edgar Y. Walker, R. James Cotton, Wei Ji Ma, and Andreas S. Tolias. A neural basis of probabilistic computation in visual cortex. Nature neuroscience, 23(1):122-129, January 2020. ISSN 1546-1722 1097-6256. doi: 10. 10.103884415995--190555555. Place: United States.
[72] Daniel L. K. Yamins and James J. DiCarlo. Using goal-driven deep learning models under- stand sensory cortex. Nature Neuroscience, 19(3):356-366 -655, Mar 2016.
[73] Huzheng Yang, James Gee, and Jianbo Shi. Brain decodes deep nets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 23030-23040, 2024.
[74] Jacob Yeung, Andrew Luo, Gabriel Sarch, Margaret M Henderson, Deva Ramanan, and Michael Tarr. Neural representations of dynamic visual stimuli. arXiv preprini arXiv:2406.0m 02659, 2024.
17

---

A Technical Appendices and Supplementary Material
A.1 Social impacts
In this work, we present a method for in silico mapping of semantic selectivity across cortical parcels in the whole brain. For cognitive neuroscience, our approach deepens the field's understanding of visual processing well beyond classical visual cortex. Clinically, deviations from normative whole-brain semantic maps may act as early biomarkers for neurological and neurodegenerative dis- orders. Furthermore, these detailed semantic atlases lay foundational groundwork for next-generation brain- computer interfaces, where fine-gret mre-grained category-level decoding could translate user's in- tended concepts-rather than low-level motor signals- into control commands for communication or assistive devices.
18

---

A.2
Reproducing selectivity for known areas
We show superstimuli from parcels overlapping with body-, place-, and word-selective areas in order to show that our pipeline can reproduce the known selectivity.
EBA
Natural Scenes Dataset
Generated
Imagenet
PPA
Natural Scenes Dataset
Generated
Imagenet
mfs-words
Natural Scenes Dataset
Generated
Imagenet
89A5
STOP
XCTEEY
IBA
biNES
Gnrl
Uu
AnA
SESO
19

---

A.3
Parcel selection process
Subject left hemisphere parcel selection (b)
Cuslling pareaa by robe celing
Quiliging parces profic on ecc.ret
Selected parcels mose calling and predictie acor way.
E.4
06
Figure S1: Selecting parcels outside visual cortex for experimentation. Parcels outside the visual area that have sufficiently high mean (a) noise ceiling and (b) model prediction accuracy chosen for further experimentation. (c) Selected parcels by mean noise ceiling and model prediction accuracy.
20

---

A.4
Reproducing known visual hierarchy properties
We optimized 32 superstimuli (16 per hemisphere) that maximally activate parcels in VI, V2, and V4 using the BrainDIVE framework. We additionally include results on FFA as representative cT-cortex ROI.
Generated superstimuli for early visual areas reproduce the classic fine-to-coarse hierarchy: V1 stimuli appear cluttered scenes filled with dense, repetitive texture; V2 images add composite color patches and rudimentary objects; V4 stimuli reveal smoother, recognizable object forms; and FFA images almost exclusively depict close-up faces, often with clear emotional expressions.
To quantify hierarchical properties, we examined whether the spatial-frequency content our stimuli mirrors classical physiological findings (e.g.. B1 (8]. We computed the radial average power spectrum for each image [70] and calculated the proportion of spectral power above 10%, 20%, and 30% of the maximum spatial frequency. At every threshold the nigh-frequeccy energy ratio decreases monotonically from V2 > V4 FFA, indicating that the images that best drive higher-level areas (FFA) contain proportionally less fine-scale texture and relatively more coarse, low-frequency structure.
Table 6: High-frequency energy ratio across ROIs at different thresholds, subject 1.
Threshold
ROI
High-frequency energy ratio
0.1
V1
0.006547
V2
0,004160
V4
0.002500
FFA
0.001347
0.2
VI
0.001838
V2
0.000944
V4
0.000699
FFA
0.000299
0.3
VI
0.000762
V2
0.000339
V4
0.000285
FFA
0.000104
Below we include superstimuli optimized for each of the above ROIs to visualize the image frequency differences.
21

---

(a) VI
(b) V2
(c) V4
(d) FFA-1
Figure S2: BrainDIVE stimuli optimized for parcels across the visual hierarchy, subject 1.
22

---

A.5 Analyzing ROI queries to map parcel connectivity
Taking advantage of our transformer-based approach, we map the connections between visual area parcels by examining the representational similarity of learned ROI queries, each of which corresponds single parcel. For both functional connectivity and ROI query similarity, we first take the full 1000 parcel x 1000 parcel matrix, then mask out the labeled area, and finally average across top three parcels that overlap with each labeled area. (Thus, not every entry on the diagonal 1.) High cosine similarity between ROI queries (queries averaged across ensemble models) suggests that two parcels are highly connected, as both attend similar content an image.
This similarity matrix for the visual areas closely replicates the functional connectivity matrix from the Schaefer parcellation (derived from resting-state correlated responses) 40 the Pearson correlation between the upper triangle of the two matrices is 2 0.5.
Below report the functional connectivity and ROI query cosine similarity matrices for subject 1.
Necel fandiom Crmecktylltte
UAGE
LH42
www
USON
UHIA
the
roen
Functional connectivity, left hemisphere
(b) ROI query similarity, left hemisphere
Query only Mark
SM2
NNV22
RI93
Mon
(c) Functional connectivity, right hemisphere
(d) ROI query similarity, right hemisphere
Figure S3: Functional connectivity matrix ROI query similarity, subject 1.
23

---

A.6 Attention maps
While one can easily make guesses about parcel selectivity simply by looking the prominent feature of the image, we can examine the attention scores from cross- cross-attention (Figure to interpret the selectivity of any parcel. The attention maps are visualized in Figure S4 for the images from Figure
Nutural Soanes Datenet
Denerated
Natural Suanaos Dateaet
Ganerated
Figure S4: Attention maps for aTL-faces and mfs-words. Yellow areas represent the highest attention weights and purple represents the lowest. Held-out NSD images are selected based on ground-truth parcel activation, but attention maps are from the encoder since no ground-truth attention maps exist. See Figure 5 for the original images.
As expected, the areas with highest attention weights for the aTL-faces parcel primarily overlap with faces, while the attention weights for the mfs- words parcel overlap with words (when present). Compared to previous work using a similar encoder model [3] our attention maps appear to follow the expected stimuli considerably less closely, possibly because our parcel boundaries were determined priori and don't 't align well with the boundaries of the actual labeled area.
24

---

A.7 Parcel locations and BrainDIVE/NSD images accompanying Figure [
(a) Parcel location in cortical flatmap.
(b) Parcel location inflated
map
(c) Held-out NSD that maximally activate the parcel.
(d) BrainDIVE generated images optimized for the parcel.
(c) Maximally-activating Ima-
geNet images.
Figure S5: Skateboarding parcel location and full collages, subject 1.
25

---

(a) Parcel location in cortical flatmap.
(b) Parcel location on inflated map
(c) Held-out NSD that maximally activate the parcel.
(d) BrainDIVE generated images optimized for the parcel.
(c) moximmall--actvvatin Ima- geNet images.
Figure S6: Child eating parcel location and full collages, subject 1.
26

---

A.8
Labeling top subject-specific parcels
For each of the parcels shown in Figure we show the top ImageNet images and the caption generated by GPT-4o (see Appendix A.18 for prompt details).
ImageNet top 4
ImageNet top
Parcel
Caption
Parcel
Caption
Sports, Team sports. Outdoor activities
Sports, Tennis, Soccer
S1 211
S5 RH 476
Extreme sports,
Soccer, Tennis, Sports
RH 373
S5 239
Skatchoarding. BMX biking
Interior Design,
Sports, Tennis, Soccer
S1 LH 238
S5 LH 367
Home Decor,
Kitchen
Babies, Children, Family
Soccer, Ice hockey, Tennis
S1 456
S5 RH 391
Kitchen, Appliances, Cabinets
Family, Parenting, Children
S1 LH 496
S5 LH 167
Food storage, Office supplics, Kitchen appliances
Tennis, Baseball, Sports
$2 LH 480
S7 RH 474
Gymnastics, Dance, Tennis
Tennis, Baseball, Sports
S2 RH 391
S7 RH 477
Tennis, Gymnastics, Basehall
Sports, Basehall, Tennis
$2 I.H 500
S7 496
Social Gatherings, Education and Learning, Sports and Recreation
Baseball, Pitching. Batting
$2 LH 496
S7 LH 497
Interior Design, Living Spaces, Furniture
Baseball, Pitching, Batting
S2 LH 365
S7 RH 317
Table 7: Maximally parcel-activating images (predicted) for unlabeled parcels.
27

---

A.9
Full collages for cross-subject tools parcel
(a) Subject
(b) Subject 2
(c) Subject 5
(d) Subject 7
Figure S7: Top 25 maximally parcel-activating ImageNet images (based on encoder predictions) for the cross-subject parcel in Figure
A.10 Statistical tests for subject-specific parcel evaluation
Here, we explain the details of the statistical tests performed in 4.4
The underlying assumption in cognitive neuroscience is that for some parcels, there exists some category of visual stimuli that maximally activates the parcel.
Our goal is assess whether the semantic content of the images encoder identifies highly activating aligns with the semantic content of NSD test images that actually elicit high ground-truth activations in that parcel. We ask whether single "concept vector" derived from encoder for a given parcel predicts -actvllactivation rank order the NSD test set more accurately than the baseline model.
We computed two concept vectors per parcel: one based on the top 32 maximally-activating ImageNet or BrainDIVE images predicted by our encoder, and one based on the top 32 maximally-activatin NSD training images. We embedded each the images into CL.P space and then averaged the CLIP vectors to obtain parcel-specific "concept vector" that reflects predicted parcel categorical
28

---

selectivity. We then embedded each NSD test image into CLIP space, and ranked images by cosine similarity to the concept vector (encoder-derived or baseline). Since we expect that higher semantic similarity to the "concept vector" will correspond to higher parcel activation, we quantified alignment with the empirical activation ordering using Spearman's 's rank correlation between concept-vector similarity rank and ground-truth fMRI activity rank.
In Table 2 for each parcel, we compared the rank correlation coefficient with a null distribution, bootstrapped with 10,000 random rankings of the NSD test images. We separate the results into most activating stimuli predicted by the model from BrainDIVE and ImageNet, as well as stimuli from the NSD training set (baseline).
A.11
Statistical tests for cross-subject parcel evaluation
For given parcel and dataset, generated selectivity hypothesis by averaging the CLIP vectors corresponding to the top 32 images according to the brain encoder, then used that CLIP vector to rank the NSD training images using cosine similarity. We calculate Spearman rank correlation between that ranking and the ranking determined by ground truth parcel activation, and compare this to a null distribution of correlation coefficients, which consists of 10,000 draws of random rankings.
Note that there are fewer number total parcels measured here since each subject has different parcels selected for experimentation (see Section 3.3), , cross-subject selectivity can only be tested for parcels selected for experimentation with every subject.
A.12 Cosine similarity analysis of labels for cross-subject parcels
To demonstrate cross-subject consistency in semantic selectivity, we also compared directly the concept vectors generated by the encoder. Concept vectors generated for the same parcel, when analyzed across subjects, tend to be more similar (0.784 = 0.069 for BrainDIVE) than the parcels coming from around the target parcel (0.710 = 0.115 for BrainDIVE). Absolute cosine similarities high because these parcels selective for high-level visual concepts that share similar features, but our method can adjudicate between them using targeted stimulus generation.
29

---

A.13 Head-to-head encoder model versus NSD train baseline
Table expands upon the results shown in table 2 reporting the fraction of parcels for which the encoder-selected stimuli outperforms the NSD train baseline in Spearman's 's rank correlation coefficient, evaluated within the same subject.
Table 8: Head to head comparison between our model and NSD train baseline: number of parcels where model rank correlation coefficient beats NSD train, evaluated within the same subject.
Comparison type
S1
S2
S5
S7
Our encoder w/ImageNet > NSD train:
93/181
94/192
89 19/175
62/ 02/196 76/196
Our encoder w/BrainDIVE > NSD train:
95/181
123/192
100/172
Table[lexpands upon results in Table reporting the number of parcels for which the rank correlation coefficient from the specified model is greater than that from the baseline model, when three subjects used to predict held-out subject.
Table 9: Head to head comparison between our model and NSD train baseline: number of parcels where model rank correlation coefficient beats NSD train, evaluated with all subjects used to predict one held out.
Comparison type
S1
S2
S5
S7
Our encoder w/ImageNet > NSD train:
30/49
35/49
36/49
27/49
Our encoder w/BrainDIVE > NSD train:
29/49
24/49
30/49
22/49
A.14
Choosing parcels shared across subjects for future fMRI experimentation
In Figure S8
we report similar results as in Figure 6 except for top parcels shared across subjects.
NSD Train Imagenet Rotricval Rato
21
15.55
20
24
Figure S8: See caption from figure for description. Each parcel shown is shared across subjects and the encoder train three subjects is used to predict the selectivity of that same parcel on held-out subject.
30

---

A.15 Interpreting statistical test results by comparing to known visual areas
To quantify what rank correlation magnitude constitutes meaningful selectivity, we report the rank correlation the model achieves on known visual areas benchmark. Parcels with known high- level selectivity (EBA, FFA, FBA) tend exhibit very high correlations, while parcels with known lower-level selectivity (V1-V4) tend to exhibit low correlations. The average Spearman's 's from our concept vectors are in the range of areas like PPA and RSC.
Using CLIP as our metric allows us to capture high-level features that can explain semantic selectivity, Given that IMRI studies tend to rely on experimenter-curatted topics, the CLIP space is move toward a data-driven approach, but ultimately any metric can be substituted.
Figure S9:
Spearman's 's actual vs. predicted activation ordering for subject 1. Top: high-level areas. Bottom: low-level areas.
(a) High-level areas
Region
ImageNet
BrainDIVE
Unlabeled parcel mean (ours)
0.168 + 0.10...6
0.163 + 0.121
EBA
0.432
0.501
FFA-1
0,217
0.243
FFA-2
0.373
0,412
FBA-2
0.373
0.401
lateral
0.323
0.339
PPA
0.166
0.128
RSC
0.170
0.162
aTL-faces
0.120
0.138
mTL-words
0.164
0.199
(b) Low-level areas
Region
ImageNet
BrainDIVE
Unlabeled parcel mean (ours)
0.168 + 101
0.163 + 0.121
V1d
0.010
0.027
Viv
0.024
0.036
V2d
-0.032
-0.035
V2v
-0.030
-0.002
V3d
-0.004
-0.018
V3v
0.007
0.009
hV4
0.047
0.067
early
-0.025
-0.014
Note that since we're re using the Schaefer- 1000 parcellation, for each visual area, we average the rank correlation value across the top 3 Schaefer parcels with greatest overlap with that visual area.
31

---

A.16 Varying top images used for statistical test
To confirm that the superior performance of ImageNet and BrainDIVE superstimuli stems from broader diversity of concepts present and not sampling near peak activation -e varied the top-K image threshold used define the concept vector. We report K 16,8, 4, 2, versions for tables 2
-woe
and 5
Table 10: Fraction of parcels whose model-derived label predicts parcel activation rankings sig- nificantly better than chance (p < 0.05, FDR corrected), varying number top images for label generation.
Model type
SI
S2
S5
S7
k=64
NSD train
134/181
156/192
134/175
154/196
Our encoder w/ImageNet
139/181
166/192
134/175
149/196
Our encoder w/BrainDIVE
128/181
168/192
140/175
152/196
k=32
NSD train
150/181
163/192
136/175
159/196
Our encoder w/ImageNet
139/181
167/199
130/175
150/196
Our encoder w/BrainDIVE
135/181
170/192
139/175
156/196
k=16
NSD train
154/181
169/192
137/17
157/196
Our encoder w/ImageNet
138/181
166/192
134/175
151/196
Our encoder w/BrainDIVE
139/181
171/192
136/175
152/196
k=8
NSD train
144/181
160/192
131/175
149/196
Our encoder w/ImageNet
138/181
162/192
134/175
153/196
Our encoder w/Bv
135/181
168/192
136/175
153/196
k=4
NSD train
134/181
146/192
122/175
140/196
Our encoder w/ImageNet
138/181
164/192
129/175
148/196
Our encoder w/BrainDIVE
140/181
166/192
142/175
151/196
k-2
NSD train
131/181
125/192
109/175
110/196
Our encoder w/lmageNet
134/181
154/192
122/175
143/196
Our encoder w/BrainDIVE
141/181
171/192
136/175
136/196
k=1
NSD train
114/181
120/192
92/175
86/196
Our encoder w/lmageNet
133/181
145/192
113/175
130/196
Our encoder w/BrainDIVE
137/181
162/192
119 175
131/196
32

---

Table 11: Average parcel-level correlations (mean + std) between model-derived labels and parcel activation rankings, varying number of top images for label generation.
Model type
S1
S2
S5
S7
k=64 NSD train
0.145 + 0.098
0.153 + 0.088
0.141 + 0.096
0.142 = 0.085
Our encoder w/ImageNet
0.167 0.106
0.163 + 0.082
0.143 0.092
0.133 = 0.075
Our encoder w/BrainDIVE
0.154 + 0.119
0.187 + 0.099
0.153 + 0.095
0.131 + 0.084
k=32
NSD train
0.162 0.098
0.164 + 0.091
0,150 0.094
0.148 = 0.086
Our encoder w/ImageNet
0.168 + 0.106
0.163 + 0.082
0.142 - 0.092
0.133 = 0.075
Our encoder w/BrainDIVE
0.163 + 0.121
0.190 + 0.099
0.154 - 0.094
0.133 = 0.083
k=16
NSD train
0.167 + 0.096
0.170 + 0.093
0.149 + 0.096
0.148 = 0.087
Our encoder w/ImageNet
0.169 0.106
0.162 + 0.082
0.142 - 0.091
0.131 = 0.075
Our encoder w/BrainDIVE
0.168 + 0.120
0.190 + 0.099
0.155 + 0.092
0.134 0.082
k=8
NSD train
0.160 0.103
0.164 + 0.094
0.140 + 0.100
0.139 + 0.091
Our encoder w/ImageNet
0.168 + 0.107
0.159 + 0.081
0.142 + 0.092
0.128 = 0.073
Our encoder w/BrainDIVE
0.167 + 0.121
0.188 + 0.099
0.155 + 0.092
0.133 + 0.082
k=4
NSD train
0.151 0.112
0.147 + 0.099
0.128 0.097
0.124 0.091
Our encoder w/ImageNet
0.165 + 0.108
0.155 + 0.082 0.184 + 0.098
0.140 - 0.091
0.124 0.073
Our encoder w/BrainDIVE
0.165 0.120
0.151 0.092
0.131 0.080
k=2
NSD train
0.134 + 0.114
0.127 + 0.106
0.110 + 0.104
0.106 + 0.092
Our encoder w/lmageNet
0.158 0.107
0.147 + 0.082
0.135 0.090
0.118 = 0.071
Our encoder w/BrainDIVE
0.160 + 0.117
0.176 + 0.094
0.149 + 0.090
0.127 = 0.080
k=1
NSD train
0.121 0.120
0.109 + 0.100
0.088 0.105
0.087 + 0.090
Our encoder w/lmageNet
0.148 + 0.101
0.136 + 0.084
0.125 + 0.091
0.112 + 0.075
Our encoder w/BrainDIVE
0.148 0.115
0.165 + 0.093
0.140 + 0.092
0.119 + 0.082
33

---

Table 12: Fraction of parcels shared across subjects whose model-derived label predicts parcel activation rankings significantly better than chance (p < 0.05, FDR corrected), varying number of top images for label generation.
Model type
S1
S2
S5
S7
k=64
NSD train
37 49
44/49
40/49
40/49
Our encoder w/ImageNet
38/49
45/49
42/49
39/49
Our encoder w/BrainDIVE
34/49
40/49
37/40
38/49
k=32
NSD train
47/49
44/49
40/49
40/49
Our encoder w/ImageNet
38/49
45/49
42/49
39/49
Our encoder w/BrainDIVE
37/49
40/49
38/49
38/49
k=16
NSD train
37 37/49
42/49
40/49
40/49
Our encoder w/lmageNet
38/49
46/49
42/49
39/49
Our encoder w/BrainDIVE
37/49
40/49
38/ 38/49
38/49
k=8
NSD train
36/49
40/49
41 41/49
40/49
Our encoder w/ImageNet
37/49
45/49
43 /3/49
39/49
Our encoder w/BrainDIVE
37/49
40/49
38/49
39/49
k=4
NSD train
40/49
41/49
40/49
38/49
Our encoder w/ImageNet
38/49
44/49
42/49
40/49
Our encoder w/BrainDIVE
37/49
40/49
39
39/49 49
41/49
k=2
NSD train
40/49
40/49
40/49
40/49
Our encoder w/lmageNet
38/49
45/49
42/49
38/49
Our encoder w/BrainDIVE
37/49
43/49
39/49
40/49
k=1
NSD train
37 /49
35/49
38/49
40/49
Our encoder w/ImageNet
37 37/49
44/49
42/49
37/49
Our encoder w/BrainDIVE
39/49
40/49
40/49
40/49
34

---

Table 13: Average parcel-level correlations (mean + std) between model-derived labels and parcel activation rankings, varying number of top images for label generation.
Model type
S1
S2
S5
S7
k=64 NSD train
0.066 + 0.068
0.092 + 0.065
0.091 + 0.073
0.090 = 0.069
Our encoder w/ImageNet
0.072 0.072
0.105 + 0.074
0.105 0.078
0.093 = 0.076
Our encoder w/BrainDIVE
0.075 + 0.076
0.102 + 0.087
0.102 + 0.090
0.087 = 0.080
k=32
NSD train
0.068 0.068
0.094 0.065
0.090 0.074
0.091 = 0.070
Our encoder w/ImageNet
0.072 + 0.071
0.105 + 0.073
0.105 + 0.078
0.093 = 0.075
Our encoder w/BrainDIVE
0.078 + 0.077
0.103 + 0.087
0.106 = 0.091
0.089 = 0.079
k=16
NSD train
0.068 + 0.067
0.093 + 0.066
0.089 + 0.072
0.090 = 0.069
Our encoder w/ImageNet
0.073 0.070
0.105 + 0.073
0.107 = 0.079
0.091 = 0.076
Our encoder w/BrainDIVE
0.081 + 0.078
0.103 + 0.087
0.106 + 0.093
0.090 = 0.080
k=8
NSD train
0.070 0.067
0.095 + 0.064
0.088 + 0.072
0.088 + 0.067
Our encoder w/ImageNet
0.071 + 0.070
0.104 0.074
0.105 0.077
0.091 = 0.075
Our encoder w/BrainDIVE
0.083 + 0.077
0.102 + 0.086
0.106 + 0.093
0.093 = 0.079
k=4
NSD train
0.068 0.065
0.093 + 0.068
0.090 + 0.072
0.086 = 0.071
Our encoder w/ImageNet
0.071 + 0.070
0.106 + 0.075
0.104 - 0.075
0.092 = 0.076
Our encoder w/BrainDIVE
0.083 0.078
0 0.086
0.110 + 0.093
0.093 0.079
k=2
NSD train
0.073 + 0.068
0.093 + 0.068
0.090 + 0.076
0.082 + 0.071
Our encoder w/lmageNet
0.070 0.068
0.108 + 0.073
0.102 0.073
0.088 = 0.073
Our encoder w/BrainDIVE
0.084 + 0.079
0.103 + 0.086
0.111 + 0.087
0.094 = 0.081
k=1
NSD train
0.062 0.069
0.079 + 0.075
0.081 0.076
0.078 0.063
Our encoder w/lmageNet
0.068 + 0.070
0.105 + 0.070
0.104 + 0.076
0.086 + 0.076
Our encoder w/BrainDIVE
0.084 0.080
0.099 0.090
0.109 0.087
0.090 = 0.082
35

---

A.17 Generating concept vectors using predicted activations
In the main text, we generated the baseline NSD train concept vector using the top maximally- activating images based on measured responses of the training set. Here, we generate concept vectors using top images based on predicted responses of the same training set-reproducinn tables 2 3 and Interestingly, the rankings generated by the model-predicted concept vector outperforms the concept vector generated by measured responses, and even our model some cases. We suspect that using model-pres -preiccpe responses may further reduce the noise compared to measured responses and be responsible for the increase in performance. We did not include the model-ppediccte concept vector performance in the main text since it's not quite baseline (since it relies on the encoding model choice).
Table 14: Fraction parcels whose model-derived label predicts parcel activation rankings signifi- cantly better than chance (p 0.05, FDR corrected)
Model type
S1
$2
S5
S7
NSD train (ground truth) NSD train (model-predicted)
150/181
163/199 192/19 176/192
136/175
155 155/155 /196
169/18) 181
156/175
168 (68/196
Table 15: Spearman's p (mean + std) between the model-predicted and ground-truth activation rankings on the NSD test set, averaged across parcels.
Model type
S2
S5
S7
NSD train (ground truth)
0.162 0.098
0.164 + 0.091
0.150 + 0.094
0.1488 + 0.086
NSD train (rodel-predicted)
0.191 0.092
0.187 + 0.084
0.174 + 0.087
0.155 + 0.082
Notes. INet = ImageNet. BD = BrainDIVE.
Table 16:
Fraction of parcels shared across subjects whose model-derived label predicts parcel activation rankings significantly better than chance (p 0.05, FDR corrected)
Model type
S1
S2
S5
S7
NSD train (ground truth)
37 /49
44/49
40/49
40/49
NSD train (model-predicted)
40/49
46/49
43/49
40/49
Table 17: Spearman's p (mean + std) between the model-predicted (from all subjects other than
heldout) and ground-trune -truh activation rankings on the NSD training set, averaged across parcels.
Model type
S1
S2
S5
S7
NSD train (ground truth)
0.068 + 0.068
0.094 + 0.065
0.090 + 0.074
0.091 + 0.070
NSD train predecepredicted)
0.078 + 0.074
0.110 + 0.068
0.113 + 0.081
0.101 + 0.074
Notes: INet = ImageNet. BD = BrainDIVE.
36

---

A.18 LLM prompts used
We used OpenAT's GPT-4o to caption the images selected by our model in order to obtain a linguistic representation of the selectivity of a parcel. The captions in Figure 7 were generated with ChatGPT 4o (04/06/2025) with collage of top-25 encoder-ranked ImageNet images shown, followed by the instruction "Describe a theme present in most of the images presented." The captions in Table 7 were generated by GPT-4o (05/12/2025, via the API) with the prompt "Give keywords for the central concept or categories present in these images."
A.19
NSD dataset quirks
Figure S10: Top 25 maximally activaaaci ImageNet images (based encoder predictions) for a parcel in subject 2, left hemisphere.
Figure $1 1: Top 25 maximally parcel-activating NSD images (based on ground truth) for a parcel in
subject 2, left hemisphere.
37

---

A.20
Compute used
We used CPU (AMD EPYC 7662), GPU (NVIDIA A40, L40), memory, and storage resources from an internal cluster. Storage for the entire project totals roughly 10TB. Training the model used roughly 3,000 GPU hours, 24,000 CPU core hours, and 32 GB per GPU hour. Running the remaining experiments used roughly 20, 000000 GPU hours, 160,000 CPU core hours, and 32 GB per GPU hour. The full project required more compute than these estimates due failed experiments, experiments not included in the paper, and model iteration.
38

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
In the abstract, we explain our approach and the results, include parcels of which we labeled the selectivity and statistical tests that verify our labels. The Section Methods explains the implementation details for our model. S Experrmmett show examples of parcels that labeled and the results from the statistical tests that show labels do drive the activation of the parcels.
Justification:
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings. It is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification: Limitations are discussed in Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to-text system might not be used reliably provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged
the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
39

---

Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Information about the datasets, model architecture, and steps to perform statistical tests are all contained in the paper.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to model. In general. releasing code and data often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
specific model and empirical evaluation, may
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (c.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing verifying the results.
40

---

reproducible, they 5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, described in supplemental material?
Question:
Answer: [Yes]
Justification: The supplementary zip file contains open access to the data and code with sufficient documentation.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.cr
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/nipss wwwpp//wwcggpuu.ccocooooooo /Codeeeeee ceabbbmssssonni swiniiossiiiiccyy for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Training and test details can be found in the supplementary zip file containing the code to reproduce the experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: Error bars and p-values are reported in all tables. Details for implementation of the statistical tests are found in the main text and appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
41

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CL, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
See Appendix A.20 for estimates on computer resources used.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics
Answer:
[Yes]
Justification:
The research conforms in every respect with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
The paper's societal impacts are discussed in Appendix A.1
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
42

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: The paper poses no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: The paper cites any data and models used and respects the respective licenses.
Guidelines:
The answer NA means that the paper does use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
43

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: Anonymized assets will be provided in the supplementary zip. Materials will include documentation.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsoureing nor research with human subjects
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
44

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [No]
Justification: The core method in this research does involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non ntn-standard components.
Please refer to LLM policy for what should or should not be described.
45


Reviews:
[{"review_id": "2uGrzaqjFE", "reviewer": "Reviewer_jfLo", "summary": "This paper introduces a novel and powerful in silico paradigm for discovering and mapping the visual selectivity of cortical regions across the entire brain. The central motivation is to overcome the limitations of traditional fMRI studies, such as high cost, small stimulus sets, and experimenter bias. The authors develop a transformer-based encoder-decoder model that learns to predict fMRI activity from natural images, leveraging a cross-attention mechanism to align deep network features with responses in specific brain parcels. By training this model on the large-scale Natural Scenes Dataset (NSD), they create a \"digital twin\" of the brain, enabling large-scale virtual experiments. This allows them to search massive image datasets (ImageNet) or use generative models (BrainDIVE) to identify or create \"superstimuli\" that are predicted to maximally activate specific parcels. The work successfully replicates known selectivity for faces, discovers novel, complex selectivity for concepts like \"skateboarding\" and \"tool use\" in previously unlabeled regions, and validates that these computationally-derived hypotheses better explain held-out brain activity than traditional baselines.", "questions": "### Questions/Clarifications for the Authors\n- The parcel-based approach is grounded in functional localization, yet the brain also relies heavily on distributed representations. While the paper notes that selectivity is \"graded\", could you further elaborate on how you view these highly selective parcels as nodes within wider, distributed networks? Does your framework offer a way to map the connections between these specialized nodes?\n- The results depend on the Schaefer-1000 functional parcellation. How sensitive do you believe the findings are to this specific parcellation scheme? Have you considered how using a different atlas, or perhaps subject-specific functional parcels, might alter the discovered patterns of selectivity?\n- The generated BrainDIVE images (e.g., Figure 9d, 10d) are a fascinating result of the pipeline. Based on your model, what is the predicted quantitative increase in activation these \"superstimuli\" would elicit compared to the best-performing images from ImageNet or the NSD test set?\n- While the parcel selection method (Section 3.3) is very well-reasoned, the general low predictive accuracy in many non-visual areas (Figure 3) remains a challenge. Do you believe this is primarily a limitation of using encoders trained on static 2D images to model regions that may process more dynamic or abstract information, or is it more related to inherent signal-to-noise issues in fMRI for these regions?", "limitations": "yes", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**###** **Strengths**\n\n**Innovative Computational Paradigm:** The paper's core contribution is a powerful data-driven framework for discovery that moves beyond traditional hypothesis-testing. The use of in silico experimentation to generate and test hypotheses on a massive scale is a significant methodological advance for the field.\n\n**Sophisticated and Effective Architecture:** The use of a transformer-based encoder with a cross-attention mechanism is a state-of-the-art approach to brain encoding. The systematic comparison of different backbones and the finding that DINOv2 provides a strong performance baseline adds to the paper's technical rigor.\n\n**Rigorous Multi-Level Validation:** The authors effectively validate their method by first confirming its ability to identify known category-selective regions (aTL-faces, Figure 4), and then by demonstrating quantitatively that their novel hypotheses for unlabeled parcels significantly predict held-out neural data better than a baseline model (Tables 2 & 3).\n\n**Discovery of Complex and Generalizable Selectivity:** The pipeline successfully identifies parcels with intriguing, complex selectivity (e.g., \"child eating,\" Figure 1c) and, importantly, shows that some of these preferences are consistent for the same anatomical parcel across multiple subjects (e.g., the \"tool parcel,\" Figure 6), greatly strengthening the biological plausibility of the findings.\n\n### Weaknesses and Possible Improvements\n**Lack of Experimental Validation for Generated Stimuli:** A primary limitation, acknowledged by the authors, is that the \"superstimuli\" generated by BrainDIVE have not yet been tested in new fMRI experiments. While the encoder-based reranking provides confidence, showing these novel images to subjects to confirm they elicit stronger-than-normal responses would be the ultimate validation of the generative pipeline.\n\n**Potential for Dataset Bias:** The model is trained exclusively on the NSD dataset, which may introduce biases. The authors astutely note a potential \"zebra-selective\" parcel as a possible artifact of the NSD stimulus set (Appendix A.12). Future work could strengthen the generalizability of the findings by training or fine-tuning the model on multiple, diverse fMRI datasets.\n\n**Minor Clarity Issues in Figures:** Some figures could be slightly improved for readability. For instance, the caption for Figure 3 describes two distinct subplots but does not use (a) and (b) labels to explicitly link the text to the corresponding images, which could cause momentary confusion.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank reviewer jfLo for recognizing the strengths of our work and for the thoughtful review. We especially appreciate the positive evaluation of our manuscript. We include answers to specific questions below.\n\n> **Q1) Mapping the parcels within a distributed network**\n\nTaking advantage of our transformer-based approach, we map the connections between parcels by examining the representational similarity of learned ROI queries, each of which corresponds to a single parcel. High cosine similarity between ROI queries (queries averaged across ensemble models) suggests that two parcels are highly connected, as both attend to similar content in an image. Indeed, this similarity matrix for the visual areas closely replicates the functional connectivity matrix from the Schaefer parcellation (derived from resting-state correlated responses) [1]; the Pearson correlation between the two matrices is $\\approx 0.5$. We will include these results and the corresponding figures in the revision.\n\nFurthermore, generated superstimuli for early visual areas reproduce the classic coarse-to-fine hierarchy: V1 stimuli appear as cluttered scenes filled with dense, repetitive texture; V2 images add composite color patches and rudimentary objects; V4 stimuli reveal smoother, recognizable object forms; and FFA images almost exclusively depict close-up faces, often with clear emotional expressions. We will include these superstimuli in the revision.\n\nTo quantify hierarchical properties, we examined whether the spatial-frequency content of our stimuli mirrors classical physiological findings (e.g., [2,3]). We computed the radial average power spectrum for each image [4] and calculated the proportion of spectral power above 10%, 20%, and 30% of the maximum spatial frequency. At every threshold the high-frequency energy ratio decreases monotonically from V1 > V2 > V4 > FFA, indicating that the images that best drive higher-level areas (FFA) contain proportionally less fine-scale texture and relatively more coarse, low-frequency structure.\n\n\n| Threshold | ROI | High-frequency energy ratio |\n|-----------|-----|----------------------------|\n| 0.1 | V1 | 0.006547 |\n| 0.1 | V2 | 0.004160 |\n| 0.1 | V4 | 0.002500 |\n| 0.1 | FFA| 0.001347 |\n| 0.2 | V1 | 0.001838 |\n| 0.2 | V2 | 0.000944 |\n| 0.2 | V4 | 0.000699 |\n| 0.2 | FFA| 0.000299 |\n| 0.3 | V1 | 0.000762 |\n| 0.3 | V2 | 0.000339 |\n| 0.3 | V4 | 0.000285 |\n| 0.3 | FFA| 0.000104 |\n\nNote: Natural-image datasets (ImageNet, NSD) do not show a clear V1 > V2 > V4 > FFA ordering because their spectra cluster around natural photo statistics, masking early-visual preferences.\n\n> **Q2) Alternative parcellations beyond Schaefer-1000**\n\nWe reported results from the Schaefer-1000 parcellation since it is commonly used in the literature, but we also ran analyses on a functional parcellation generated by grouping voxels with correlated responses (applying k-means on the training set, which yields a subject-specific parcellation). For each subject, the whole brain was separated into roughly 1000 parcels, each containing 200–400 voxels based on correlated responses. We observed very similar results when applying the same analyses. Parcels that overlap significantly with known visual areas demonstrate the selectivity of that area, reproducing the sanity check in 4.3, including face-, place-, word-, and body-selective areas. Many visually-responsive parcels outside the visual area also demonstrated consistent semantic selectivity of complex concepts similar to those observed in the Schaefer-1000 parcellation (social interactions, specific sports, family, etc.). Perhaps surprisingly, there were some concepts that parcels were selective for that appeared in one parcellation but not the other—though we only observed this qualitatively. These results ultimately were not included in the final manuscript for brevity and because Schaefer-1000 is a more widely-used parcellation approach.\n\n> **Q3) Quantitative increase in activation by superstimuli**\n\nOn average, our model predicts that the BrainDIVE-optimized superstimuli activate the parcel more strongly than top NSD test images, but not more than the best-performing ImageNet images. But for roughly a third of parcels, BrainDIVE-optimized superstimuli do outperform ImageNet, suggesting that some parcels may be selective for a concept that is not easily captured by the space of natural images (ImageNet).\n\nHere, we report the average activation magnitude predicted by the model for NSD-test images, top ImageNet images, and top BrainDIVE images.\n\n**Model-predicted activation magnitude for top-K superstimuli from all datasets, averaged across parcels, subject 1.**\n||Top 32, Visual Areas|Top 16, Visual Areas|Top 4, Visual Areas|Top 32, Non-Visual Areas|Top 16, Non‑Visual Areas|Top 4, Non‑Visual Areas|\n|---|---|---|---|---|---|---|\n|**NSD test**|0.476 ± 0.210|0.524 ± 0.223|0.597 ± 0.244|0.403 ± 0.188|0.475 ± 0.204|0.580 ± 0.237|\n|**ImageNet**|**0.780 ± 0.283**|**0.792 ± 0.287**|**0.813 ± 0.293**|**0.760 ± 0.265**|**0.776 ± 0.268**|**0.803 ± 0.273**|\n|**BrainDIVE**|0.737 ± 0.294|0.760 ± 0.299|0.794 ± 0.307|0.672 ± 0.291|0.711 ± 0.293|0.767 ± 0.299|\n|---|---|---|---|---|---|---|\n|**# parcels where BrainDIVE > ImageNet**|51 / 162|59 / 162|68 / 162|28 / 181|34 / 181|50 / 181|\n\n> **Q4) Low prediction accuracy in non-visual areas**\n\nSince many of the parcels considered here are deeper in the processing hierarchy, they are selective for more abstract concepts and are driven not only by a single sensory input but also multi-modal inputs and other cognitive processes (such as attention and memory). As a result, the brain responses do not always have high agreement for different presentations of the stimuli, leading to lower signal-to-noise (SNR) ratios. For this reason, we only selected parcels with an SNR above a certain threshold for further experimentation, to make sure their visual responses are strong and consistent and ensure generalizability of the observed categorical selectivities to future experiments.\n\n> **Q5) Typo in figure 3**\n\nThank you, we have added (a) and (b) labels for Figure 3 in the revision of our work.\n\n[1] Ru Kong, Yan Rui Tan, Naren Wulan, Leon Qi Rong Ooi, Seyedeh-Rezvan Farahibozorg, Samuel Harrison, Janine D. Bijsterbosch, Boris C. Bernhardt, Simon Eickhoff, and B. T. Thomas Yeo. Comparison between gradients and parcellations for functional connectivity prediction of behavior. NeuroImage, 273:120044, June 2023. doi:10.1016/j.neuroimage.2023.120044. URL\n\n[2] David H. Hubel and Torsten N. Wiesel. Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. The Journal of Physiology, 160(1):106–154, January 1962. doi:10.1113/jphysiol.1962.sp006837. URL\n\n[3] R. Desimone and S. J. Schein. Visual properties of neurons in area V4 of the macaque: sensitivity to stimulus form. Journal of Neurophysiology, 57(3):835–868, March 1987. doi:10.1152/jn.1987.57.3.835. URL\n\n[4] A. van der Schaaf and J. H. van Hateren. Modelling the power spectra of natural images: Statistics and information. Vision Research, 36(17):2759–2770, September 1996. doi:10.1016/0042-6989(96)00002-8. URL"}, {"review_id": "s6yixq2T8n", "reviewer": "Reviewer_yocx", "summary": "In this work, the authors propose a data-driven in silico approach based on fMRI activation predictive encoding models that would allow for images predicted to yield high activations in the target area to be selected from a pool of images or be generated by using a diffusion-based decoder employing the encoder model’s gradient. Using this approach, the authors higlight that the method can indeed select out mostly face-containing images for aTL (area known for face selectivity) and further show that images selected by their approach tends to yield semantically meaningful cluster (as characterized by CLIP embedding) such that images closer to this embedding in the CLIP space tends to elicit strong response in the encoding model.", "questions": "* The authors seem to be using the encoding model almost exclusively for generating the initial set of labels. In assessing the “category” defined by these images, the authors rely on the similarity of the concept vector for each image in CLIP space. However, as they have direct access to the encoding model, couldn’t they use the encoding model to yield similarity from the model? It would also be interesting to see if a similarity metric defined on the encoding model activations could better assess the rankings. \n* Instead of just generating the most activating images, could one use the encoding model to find the most distinctive set of images (e.g., images that only selectively activate the target parcel while eliciting little to no activations in other parcels)? Such a contrastive approach in semantic assignment across multiple parcels at once may help further differentiate the selectivity of each parcel.", "limitations": "Authors identify key limitations", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Quality\n* It appears to me that the encoding model-based stimulus selection as put forward by the authors in this work is conceptually very similar, if not identical, to what has been proposed and worked on over the last couple of years across electrophysiological and optophysiological data in NHP and mice. However, citations of this relevant literature appear to be grossly lacking.\n* The work offers very little novel technical contributions. While it would have been exciting to see whether the new image category selected for each parcel to be maximally activating would have actually led to higher activation, the work does not include experimental verification (the authors note this limitation). As the technique is not particularly novel (aside from possibly the technique being applied to fMRI dataset, rather than electrophysiolgy or optophysiology recordings), it would have been seminal to see the predictions tested in real experiments.\n* In the evaluation of the labels, the authors compare the degree to which the model-derived label predicts (ranks) parcel activation rankings better than chance as a measure of labeling consistency. Although the reasoning is intuitive, it is not clear whether the use of semantic proximity is always the most appropriate measure to assess the similarity of a test image to the labeled most activating image. One would imagine an appropriate measure of image similarity precisely depends on the exact visual features that the parcel is selective to, and as such, how well the semantic proximity (in their case, the proximity in the CLIP embedding) can assess the parcel-relevant feature similarity entirely depends on a priori unknown alignment of CLIP embedding and correspoinding image embedding based on the parcel activities. Furthermore, although encoder-based selections do appear to achieve higher fraction scores, the difference from the baseline is rather small. Without experimental verification, it is unclear if the baseline selection vs encoder-based selection would have yielded experimentally observable and meaningful difference in the activations.\n\n\nClarity\n* The term NSD was introduced before the first mention of Natural Scene Dataset appearing in L121 and even then explicit correspondence of the acronym was never made. Please be sure to introduce the full term at the first use of the acronym appearing on L112\n\n\nSignificance & Originarity \n* As the authors state, the use of large and naturalistic stimulus sets (such as naturalistic images) is critical in adequately studying the functions of stimulus-responsive neural populations. However,  the large number of trials required to present such datasets proves to be empirically prohibitive. In line with the recent trend, the authors identify AI models trained on an extensive collection of neural datasets as “digital twins,” which enables extensive in silico experiments. In this work, the authors introduce a pipeline framework through which a trained encoding model can be used for mapping visual categorical selectivity, which in turn can be used to generate hypotheses at the category/concept level.\n* While the proposed apporach of using a high-accuracy encoder model of the brain (in this work’s case, predicting vertices activation from image input) in selecting (from a held out natural images) or generating (based on generative model where the generation can be guided via gradiet of the predictive model) images that would most strongly activate the target brain areas is a powerful technique, this framework is by no means original as there has been numerous work exactly on this approach in the last ~6 years including the work of Bashivan, Kar & DiCarlo Science 2019, Walker, et al. NatNeuro 2019, and Perzchlewicz, et al. NeurIPS 2023, just to name a few.\n* Consequently, I’m not clear what particular novel contribution is made by the work. I fail to see the idea of using encoding model to predict highly activating stimulus to be conceptually novel (as this has been proposed with rigorous validation in the work cited above), and use of generative model guided by encoding model to generate high activating stimuli has also been addressed in the work of Perzchlwicz et al (NeurIPS 2023) with even experimental verification. Although it is possible that these techniques have not been used extensively in fMRI settings, I would find the difference in target neural recording modality alone to be sufficient grounds for considering this work novel.", "was_revised": true, "final_justification": "Authors have addressed my main concern over the connection to the prior work and clarification of the main contributions of this work.", "rebuttal": "We thank Reviewer yocx for the constructive review and for bringing our attention to missing citations for relevant works. We regret not citing them in the original submission. These papers have been seminal in laying the groundwork for in silico brain mapping and the foundation of our methodology, and we will include citations to these papers in our revision. However, we believe that our work significantly expands upon past work and clarify the novelty and contribution of our approach below.\n\n> **Clarification of a detail in the review**\n\nReviewer yocx wrote \"images selected by their approach tends to yield semantically meaningful cluster (as characterized by CLIP embedding) such that images closer to this embedding in the CLIP space **tends to elicit strong response in the encoding model**.\"\n\nWe just want to clarify that our images closer to this embedding in the CLIP space tends to elicit strong response in the ***ground-truth*** held-out set, not only by the encoding model—which allows us to validate the robustness of our predictions in silico without encoder bias.\n\n> **Q1) Technical contributions**\n\nWe share your optimism about the promise of AI models for in silico experimentation to study neural populations. However, past approaches face several key limitations that our work addresses, as pointed out by other reviewers.\n\n* Reviewer **jfLo** highlights the \"massive scale\" upon which our pipeline generates and tests hypotheses as a \"significant methodological advance for the field.\"\n* Reviewer **uJqz** cites cross-subject generalization in evaluation as a key strength of the method.\n* Reviewer **kyMo** raises the capability to identify selectivity outside canonical visual areas as a key \"proof-of-concept\" result.\n* Reviewer **ywAe** writes that the use of encoder-decoder transformers to synthesize optimal stimuli from massive datasets \"suggests a new approach for discovering visual selectivity, offering a more flexible and scalable alternative to traditional fMRI experiments.\"\n\nBelow, we address the novelty of our manuscript in comparison to the specific papers mentioned.\n\n> **Q2) Significance compared to past work in NHP/mice**\n\nWe thank the reviewer for pointing out our blindspot regarding similar works done in mice/NHP and regret not citing these studies. We believe those works laid the foundation for much of the more recent work in human imaging and we have updated the revision to include the relevant citations and past methods. Those works [1, 2, 3] have clearly demonstrated the usefulness of stimulus optimization and how they can be paired with encoders to study neural populations. The generated \"superstimuli\" also have been experimentally validated, which is a motivation for in silico experimentation that we have explored in this work.\n\nWe do however want to emphasize that extending those works to human neuroimaging comes with many new challenges that remain unaddressed. Namely, building a scalable state-of-the-art encoding model for the whole brain, using the encoder for exploring massive image sets and generative models for maximally activating stimuli, and creating robust methods to quantify selectivities and to validate them empirically in silico—have all not yet been explored. Notably, building a linear encoding model for the whole brain with the same size feature maps would have 242B parameters.† Our transformer-based model learns parcel queries that reduce feature dimension to 768 while still outperforming the full linear model. An experiment comparing the two isn't feasible to conduct but prior work comparing performance on the visual area supports this claim [4].\n\nThese innovations move the field from single‑site discovery ([1] and [3] looks at primate and macaque V4; [2] looks at monkey V1) to systematic whole‑brain mapping and to statistical validation without additional scanning time, which, to our knowledge, has not been demonstrated previously.\n\nAs a result of this undertaking, we have discovered visual selectivities to high level concepts that are unique to humans, such as sports and complex tool use, that could not have been studied in non-human animal models. We believe addressing all these challenges, and the discovery of new selectivities worthy of future experiments, are all novel aspects of our work in comparison to the papers highlighted by the reviewer.\n\n†31\\*31 (patch size) \\* 768 (dimension size) \\* 327684 (voxels) = 242B parameters\n\n> **Q3) Similarity metric defined on encoding model**\n\nWe agree that we would be able to use encoder predictions to create a similarity measure, and that would be a good test of the encoder performance. However, the main goal of our methodology is to create labels for the selectivity of unlabeled parcels. Since the CLIP space provides an aligned visual-text embedding space, creating our \"concept vector\" hypothesis in that space can give us a semantic description of the selectivity. This also introduces a bottleneck that can lead to findings that can better generalize to future in vivo experiments. \n\n> **Q4) Contrastive approach for generating superstimuli**\n\nThis is an interesting suggestion for further teasing apart the selectivity of a parcel. We performed some further analyses on parcels in the labeled area, and found that even without any selection, stimuli which activate one area generally do not activate other (non-related) areas. For example, the top 32 ImageNet images that maximally activate aTL-faces already perform poorly in activating other parcels such as PPA (top 82.3%), OPA (top 81.5%), and RSC (top 62.3%).\n\nIn some cases, this approach may actually make it more difficult to correctly label the selectivity of an area. The images that maximally activate aTL-faces unsurprisingly also activate FFA (top 23.6%) and OFA (top 9.4%). Naïvely applying the suggested approach, we can generate images that activate aTL-faces but not FFA, which end up being non-face images—not the expected selectivity for aTL-faces. Of course, for labeled parcels, we could manually avoid suppressing the activation in other related areas when choosing optimal stimuli, but this fails for areas outside the visual cortex since they are unlabeled.\n\nGenerally, we believe that this approach may be useful for experimenters interested in very fine-grained analyses of specific parcels, but it is challenging to apply to whole-brain mapping. We will include these results and details about this suggested approach in the revision.\n\n> **Q5) NSD acronym typo**\n\nThank you, we revised our paper to use the full name upon first mention.\n\n[1] Pouya Bashivan, Kohitij Kar, and James J. DiCarlo. Neural population control via deep image synthesis. Science, 364(6439):eaav9436, May 2019.\n\n[2] Edgar Y. Walker, R. James Cotton, Wei Ji Ma, and Andreas S. Tolias. A neural basis of probabilistic computation in visual cortex. Nature Neuroscience, 23(1):122–129, January 2020. ISSN 1097‑6256, 1546‑1726.\n\n[3] Paweł A. Pierzchlewicz, Konstantin F. Willeke, Arne F. Nix, Pavithra Elumalai, Kelli Restivo, Tori Shinn, Cate Nealley, Gabrielle Rodriguez, Saumil Patel, Katrin Franke, Andreas S. Tolias, and Fabian H. Sinz. Energy Guided Diffusion for Generating Neurally Exciting Images. Advances in Neural Information Processing Systems, 36, 2023.\n\n[4] Hossein Adeli, Minni Sun, and Nikolaus Kriegeskorte. Transformer brain encoders explain human high‑level visual responses, 2025."}, {"review_id": "zXn1DrD89V", "reviewer": "Reviewer_uJzq", "summary": "This paper proposes a pipeline for characterizing semantic selectivity across the entire cortex using fMRI responses to natural images. The method relies on a brain encoder that predicts voxel-wise responses from images, from which a “concept vector” is constructed per cortical parcel by averaging CLIP embeddings of top-activating images. Semantic selectivity is then evaluated via Spearman correlation between parcel activation predictions and CLIP-based similarity scores. The approach is used to compare different image sources (NSD-train, ImageNet, BrainDIVE), and is extended to assess cross-subject generalization of semantic selectivity.", "questions": "Generally, I think the work has potential and raises interesting ideas, but in its current form I have concerns about its conceptual clarity and novelty. If the authors can address the following questions, I’d be happy to reconsider and possibly increase my score.\n\n**1. Could the result in Table 2 simply reflect differences in input sampling density around activation peaks, rather than concept coverage?** in Table 2, the only difference between conditions is the \"concept vector\" used per parcel, computed as the average CLIP embedding of the top-K (K=32) activating images. Since a good concept vector depends on sampling near peak activation, image sets like BrainDIVE (explicitly optimized for this) and ImageNet (large and dense) naturally yield better vectors. In contrast, the NSD-train set is relatively small and not optimized for activation maximization, making it less likely to produce **representative concept vectors** — even if the underlying concepts are present. Thus, the performance gap may reflect sampling effectiveness rather than a difference in the actual set of concepts represented. The current analysis seems to conflate these, interpreting lower values for NSD-train as indicating fewer covered concepts. To support this claim more rigorously, it would help to control for sampling density — for example, by varying the top-K threshold used to define the concept vector. If NSD-train still underperforms at lower K (where the top images should be closest to the activation peak), that would strengthen the argument.\n\n**2. Same concern applies to Table 3**: Since the encoder-driven image sets yield higher spearman rank correlations the authors conclude that “brain-encoder pipeline can generate finer-grained categorical hypothesis” than images in the train set. Again, this is very likely a direct consequence of having a better estimate of a parcel’s concept vector, which itself depends on having enough images sampled around maximum activation for that parcel. Showing these results across different values of K (i.e., the number of top-activating images used to define the concept vector) would clarify whether the NSD-train set truly yields a worse selectivity hypothesis, or if the observed differences are simply due to fewer or less optimized samples contributing to the concept vector. This would help disentangle model differences from sampling effects.\n\n**3. What are some examples of follow up experiments for which the parcels with higher retrieval in Figure 5 are promising targets?** The metric rewards parcels where the concept vector retrieves the ground-truth top-activating images well. While this likely reflects some degree of semantic selectivity, it's not clear why this makes these parcels \"promising targets\" for follow-up fMRI studies. What would an experimenter test for with these parcels and/or the corresponding images that they seem to be semantically selective for (from ImageNet or BrainDIVE)?\n\n**4. Did the NSD-train condition use actual measured responses to select the top-K images, or were the selections based on model-predicted responses to training images (like ImageNet and BrainDIVE)?** This detail is important for interpreting the NSD-train condition as a proper baseline. If the encoder is also used for NSD-train, then all conditions reflect model-based estimates. If not, then the performance gap could partly reflect differences in response quality (e.g., measured vs. predicted) rather than conceptual coverage alone. Clarifying this would help in comparing conditions more fairly.\n\n**5. What is the main conceptual advancement beyond implementing an evaluation framework for comparing image sets?** The pipeline offers an approach to quantify semantic selectivity. The main message, as far as I understood, is that since the approach relies on highly exciting inputs to construct a parcel-specific “concept vector” using a good set of inputs matter. And the findings largely align with what one would expect: images optimized to be maximally activating (BrainDIVE) or images picked from a densely sampled set (ImageNet) naturally yield better performance than a limited training set. In addition, the paper also assesses cross-stimuli and cross-subject generalization, but I find those results hard to interpret (explained in the next question). Could the authors clarify which aspects, beyond the application of existing tools, constitute a novel conceptual or methodological contribution. Is it the use of CLIP to create “concept vectors” for parcels where their semantic selectivity is not known?\n\n**6. Does the correlation magnitude matter?** In both Tables 3 and 5, the reported Spearman correlations are rather low. Assuming the concept vectors are accurate and CLIP space reflects semantic similarity, strong semantic selectivity should yield high rank alignment (i.e., high $\\rho$). However, the paper provides no principled threshold or theoretical grounding for what magnitude of $\\rho$ reflects meaningful selectivity. How high should the correlation be for a parcel to be considered semantically selective, and how low for it to be weakly/not selective? As a reference point, it would be informative to show the same analysis on the known visual parcels - do they exhibit (noticeably) higher correlations?\n\n**7. Why not directly compare concept vectors of the same parcel across subjects to demonstrate cross-subject consistency in semantic selectivity?** While Tables 4 and 5 demonstrate that concept vectors derived from other subjects exhibit semantic selectivity in a target subject, this provides only indirect evidence for cross-subject generalization. It does not directly assess whether the semantic selectivity of corresponding parcels is similar across individuals. In Figure 6, the authors illustrate this point qualitatively using example images, but a more direct test would be to compute the pairwise cosine similarity between concept vectors of corresponding parcels across subjects. This would more explicitly support the assumption that the same parcel across subjects encodes similar semantic content. Have the authors performed such an analysis?\n\nOther (minor) points:\n\n- In section 4.5 how are the authors exactly constructing the “concept vector” using the encoder models trained on other subjects? Are you pooling top-32 maximally activating images across three models (i.e. those trained on other subjects)? Please provide some details on how the concept vector is constructed.\n- Is the encoder part of the pipeline and is the encoder a contribution of the paper? My initial understanding was not the pipeline is additional steps around the encoder, which aligns well with the statement in line 64: \"...our pipeline is ultimately encoder-agnostic, and can use any encoder that is image-computable\". But then the usage of the term “brain encoder pipeline” in line 224 seem to imply that the encoder is part of the pipeline. I would appreciate some clarification here, and I would also suggest to improve the text to potentially avoid such confusions.\n- Please refer to specific subplots/panels in Figure 3 both in the text (lines 141-144) and the caption. Also, please label the colorbars.\n- There is a repeated sentence in lines 222-223.", "limitations": "The authors already point out some relevant limitations such as the need for experimental validation of “superstimuli” and the fact that encoding model can be biased due to the training set. I think these are fair and useful to acknowledge. A few more points might be worth considering:\n\n- The approach assumes that CLIP embedding space reflects something close to the brain’s semantic organization. That might hold in some regions (e.g., visual cortex), but it’s unclear how valid that assumption is more broadly.\n- The method is heavily based on top-K activation which could be problematic. For example, differences in performance across datasets could just reflect sampling density near activation peaks, not actual differences in concept coverage.\n- The Spearman correlation values used throughout (especially in Tables 3 and 5) are fairly low, and it’s unclear what values should be considered meaningful? Some sort of benchmark (e.g. visual cortex parcels) would help give these numbers context.\n- Tables 4 and 5 suggest cross-subject generalization, but they don’t directly test if the *same* parcels across subjects are encoding similar concepts. Comparing concept vectors for the same parcel across individuals (e.g. using cosine similarity) would provide a more clear evidence here.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n- Technically sound integration of encoding models and CLIP-based embeddings into a pipeline to assess semantic selectivity.\n- Explores semantic selectivity beyond canonical visual areas, using the large-scale NSD dataset.\n- Encoder-agnostic framework with potential applicability to other predictive models (although not demonstrated).\n- Evaluation includes cross-subject generalization, which strengthens the methodological contribution.\n\n**Weaknesses**\n\n- At times I found it hard to follow the methodological setup, particularly regarding how the concept vectors were constructed in different conditions (e.g., for cross-subject experiments), whether actual or predicted responses were used for NSD-train, and what exactly the authors consider part of the \"pipeline\". These ambiguities made it difficult to interpret the core comparisons and contributions.\n- Reported Spearman correlations are low, and no threshold is provided to interpret what constitutes meaningful semantic selectivity.\n- It is not clear whether the differences observed across image sets is actually due to concept coverage or could it be due to sampling density around activation peaks.\n- Several implementation details (e.g., whether measured or predicted responses were used, how concept vectors are constructed for the cross-subject analysis) were not clear to me.\n- Conceptually, the work builds on existing ideas and may offer limited novelty beyond assembling and evaluating known components.", "was_revised": true, "final_justification": "The authors provided clarifications and additional analyses that addressed my main concerns, particularly regarding sampling effects and response variability. While some questions remain, I find the conceptual framing and empirical results good enough to warrant an increased rating.", "rebuttal": "We are very grateful to Reviewer uJzq for the thorough and constructive feedback. We report the results from additional experiments below to address specific comments, and have included these results (and the motivation) to the revision. We look forward to further discussion.\n\n> **Q1) Table 2: Concern about sampling density around activation peaks as a confound**\n\nTo confirm that the superior performance of ImageNet and BrainDIVE superstimuli stems from a broader diversity of concepts present—and not sampling near peak activation—we varied the top-K image threshold used to define the concept vector as suggested.\n\nLowering $K$ from 32 to 1 yields results that closely mirror Table 2 in the paper, with BrainDIVE and ImageNet outperforming the NSD train concept vector by a similar margin. We observe the same pattern for $K=16,8,4,2,1$ but only report $K=4,1$ for brevity. All tables will be added to the revision.\n\n**Table 2 (revised).** Fraction of parcels where the superstimulus selection process ranks NSD test images better than chance, FDR corrected, $K=4,1$.\n\n$k=4$\n||s1|s2|s5|s7|\n|-|-|-|-|-|\n|NSD train|122/181|123/192|112/175|119/196|\n|ImageNet|138/181|163/192|128/175|**151/196**|\n|BrainDive|**140/181**|**165/192**|**141/175** |151/196|\n\n$k=1$\n\n||s1|s2|s5|s7|\n|-|-|-|-|-|\n|NSD train|103/181|97/192|89/175|81/196|\n|ImageNet|131/181|145/192|113/175|128/196|\n|BrainDive|**136/181**|**162/192**|**120/175** |129/196|\n\nWe would also like to note that traditional large-scale experiments are not—and cannot be—optimized for activation maximization. NSD is already the most massive fMRI image dataset to date, yet the cost of collecting data on an even larger image dataset like ImageNet is (currently) prohibitive. The greater density of concept coverage is still inherently an advantage of our in silico approach.\n\n> **Q2) Table 3: Same concern**\n\nWe ran the same experiment reported in Table 3 for different values of the top-K threshold. Lowering the top-K threshold from 32 to 1, the results are again very similar, with ImageNet and BrainDIVE outperforming NSD train by a similar margin for all $K=16,8,4,2,1$. All tables will be included in the revision.\n\n**Table 3 (revised).** Spearman's $\\rho$ (mean $\\pm$ std) between the model-predicted and ground-truth activation rankings on the NSD test set, averaged across parcels, $K=4,1$.\n\n$k=4$\n||s1|s2|s5|s7|\n|-|-|-|-|-|\n|NSD train|0.130 ± 0.119|0.117 ± 0.096|0.115 ± 0.106|0.111 ± 0.091|\n|ImageNet|**0.165 ± 0.108**|0.155 ± 0.082|0.140 ± 0.091|0.124 ± 0.073|\n|BrainDive|**0.165 ± 0.120**|**0.184 ± 0.098**|**0.151 ± 0.092**|**0.131 ± 0.080**|\n\n$k=1$\n||s1|s2|s5|s7|\n|-|-|-|-|-|\n|NSD train|0.104 ± 0.124|0.090 ± 0.095|0.092 ± 0.114|0.078 ± 0.089|\n|ImageNet|**0.148 ± 0.101**|0.136 ± 0.084|0.125 ± 0.091|0.112 ± 0.075|\n|BrainDive|**0.148 ± 0.115**|**0.165 ± 0.093**|**0.140 ± 0.092**|**0.119 ± 0.082**|\n\n> **Q3) Examples of follow-up experiments**\n\nParcels in Fig. 5 are just one way to select promising targets for follow-up fMRI studies. We suggest that experiments on these parcels would easily allow us to verify our method, to ensure that our selectivity labels are not just an artifact of the dataset or model. For these parcels, a single semantic concept (i.e., a vector in CLIP space) seems to capture the selectivity well—similar to the visual area. A simple follow-up experiment would be to present these \"superstimuli\" to a subject and check whether higher activation is observed in these parcels.\n\nIn addition, these parcels are promising since 1) they are visually responsive, suggesting they play a role in visual processing and it is of interest to define what these regions do, 2) they demonstrate consistent selectivity that our model can predict well (Figure 5), 3) their selectivity is not well understood nor mapped in past experiments since they lie outside the visual area, 4) many selectivities (see A.5: specific sports, kitchen appliances, social gatherings) are not known to have specialized regions, so it would be of interest to experimentally confirm.\n\nHowever, this is just one method of selecting \"promising\" parcels, and an fMRI experimenter could apply their own criteria to select parcels using our method. For example, an experimenter interested in tool use specifically could use our in silico method to discover parcels involved exclusively in tool use, then perform follow-up experiments to test parcel activation on the superstimuli that our pipeline generates.\n\n> **Q4) Clarification on the NSD-train condition**\n\nThe NSD-train condition uses actual measured responses to select superstimuli, while the ImageNet and BrainDIVE images use model-predicted responses. Therefore, the NSD-train condition reflects the traditional fMRI model, where an experimenter generates selectivity hypotheses based on images that experimentally maximally activate a parcel. Since all conditions are eventually evaluated on actual responses from the NSD test set, our test is, if anything, biased in favor of the NSD-train condition. That our ImageNet/BrainDIVE-based superstimuli selection still outperforms is strong evidence of the usefulness of our pipeline.\n\nThis is an important clarification that we will include in our revision.\n\n> **Q5) Clarifying main conceptual advancement**\n\n1) **Massive scale in silico experimentation:** \"Superstimuli\" from ImageNet and BrainDIVE outperform the NSD training set on predicting parcel activation on a held-out image set, demonstrating that our pipeline can successfully test parcel selectivity beyond those explicitly shown to the participant. To the best of our knowledge, no other study has tested image datasets on this scale nor demonstrated the capability to test concepts beyond the training set in silico.\n2) **Mapping of the whole brain:** Previous work has focused on areas in the visual cortex (mostly V1-V4, some limited higher-level areas). We expand semantic selectivity mapping to the whole brain beyond the visual area. We discover areas with novel complex semantic selectivity beyond the classic visual area.\n3) **In silico verification framework:** Our pipeline can verify selectivity hypotheses in silico by evaluating how well a label can predict ground-truth activation on a held-out set.\n4) **New fMRI experimental paradigm:** As both image datasets and encoding models improve, our pipeline offers a way to leverage these advances to accelerate and improve the accuracy of whole-brain mapping.\n\n> **Q6) Interpreting rank correlation magnitude**\n\nParcels in the visual area with known high-level selectivity (EBA, FFA, FBA) tend to exhibit very high correlations, while parcels with known lower-level selectivity (V1–V4) tend to exhibit low correlations. The average Spearman's $\\rho$ from our concept vectors are in the range of areas like PPA and RSC—both widely studied and accepted in the literature.\n\nUsing CLIP as our metric allows us to capture high-level features that can explain semantic selectivity. Given that fMRI studies tend to rely on experimenter-curated topics, the CLIP space is a move toward a data-driven approach, but ultimately any metric can be substituted.\n\nWe provide baselines from known areas to contextualize our results. Since selectivity is a graded response and not binary, we don’t believe it is appropriate to have a minimum threshold that constitutes meaningful selectivity. Some parcels are naturally more visually responsive than others, and our paper lays out a method of systematically making and ranking semantic selectivity predictions. Notably, the CLIP metric allows us to rank parcels by how well we can make such predictions, which paves the way for in vivo fMRI studies of promising parcels.\n\nSpearman's $\\rho$, actual vs. predicted activation ordering in high level areas, subject 1:\n\n||Imagenet|BrainDIVE|\n|-|-|-|\n|Unlabeled parcel mean (ours)|0.168 ± 0.106|0.163 ± 0.121|\n|EBA|0.432|0.501|\n|FFA-1|0.217|0.243|\n|FFA-2|0.373|0.412|\n|FBA-2|0.373|0.401|\n|lateral|0.323|0.339|\n|PPA|0.166|0.128|\n|RSC|0.170|0.162|\n|aTL-faces|0.120|0.138|\n|mTL-words|0.164|0.199|\n\nSpearman's $\\rho$, actual/predicted activation ordering in low level areas, subject 1:\n\n||Imagenet|BrainDIVE|\n|-|-|-|\n|Unlabeled parcel mean (ours)|0.168 ± 0.106|0.163 ± 0.121|\n|V1d|0.010|0.027|\n|V1v|0.024|0.036|\n|V2d|-0.032|-0.035|\n|V2v|-0.030|-0.002|\n|V3d|-0.004|-0.018|\n|V3v|0.007|0.009|\n|hV4|0.047|0.067|\n|early|-0.025|-0.014|\n\nNote that since we’re using the Schaefer-1000 parcellation, for each visual area, we average the rank correlation value across the top 3 Schaefer parcels with greatest overlap with that visual area.\n\n> **Q7) Cosine similarity analysis of concept vectors for cross-subject parcels**\n\nGreat suggestion. We do find that the concept vectors generated for the same parcel, when analyzed across subjects, tend to be more similar (0.784 ± 0.069 for BrainDIVE) than the parcels coming from around the target parcel (0.710 ± 0.115 for BrainDIVE). Absolute cosine similarities are high because these parcels are selective for high-level visual concepts that share similar features, but our method can adjudicate between them using targeted stimulus generation. We will add this analysis to give further intuition on the concept space of the parcels.\n\n> **Q8) Clarifying construction of concept vector in cross-subject analysis**\n\nYes, the concept vector for cross-subject experiments is generated by taking the simple average of the CLIP vectors for the top-32 maximally activating images from each subject. For any subject, this concept vector is an average of 96 CLIP vectors (32 per subject * 3 subjects).\n\n> **Q9) Ambiguity of the term \"pipeline\"**\n\nThe application of this encoder to the whole brain is a contribution of the paper (section 4.2), but you’re right that the core contribution is the steps around the encoder. Thank you for pointing out the ambiguity; we will clarify this in our revision.\n\n> **Q10) Typos in figure and text**\n\nThank you for pointing these out. We will fix these in our revision."}, {"review_id": "iMmFmFiGYN", "reviewer": "Reviewer_ywAe", "summary": "This manuscript demonstrates an in-silico approach for mapping visual categorical selectivity across the whole brain, moving beyond traditional methods that rely on controlled stimuli and linear encoding models. By leveraging an encoder-decoder transformer with a brain-region to image-feature cross-attention mechanism, the proposed architecture flexibly aligns high-dimensional deep network features with cortical responses. Coupled with diffusion-based image generative models and large image datasets, this method can synthesize and select images that maximize the activation of different cortical parcels, revealing regions with complex compositional selectivity involving diverse semantic concepts. The findings suggest a new approach for discovering visual selectivity, offering a more flexible and scalable alternative to traditional fMRI experiments for hypothesizing and testing visual concepts.", "questions": "**Major concerns**\n\n1. Although this research provides an interesting approach to finding superstimuli, the lack of empirical validation of these 'superstimuli' through new fMRI experiments is a clear limitation. However, I do not consider this an essential prerequisite for the paper's acceptance. To further bolster the model's persuasiveness without immediate empirical validation, more robust evidence is needed. I believe the paper's convincingness would be significantly enhanced by providing richer examples akin to Figure 4 (Verifying the selectivity of aTL-faces). Could similar examples be generated for other previously identified fMRI-sensitive areas such as the PPA (Place-selective), EBA (Body-selective), and VWFA (Word-selective)?\n\n2. Beyond merely predicting categorical selective areas, is it possible to analyze changes in characteristics along the visual hierarchy? For instance, if 'superstimuli' could be identified along the ventral visual stream from V1 to IT, and these demonstrated previously known hierarchical processing properties, it would offer another robust validation point for the model.\n\n3. The paper claims to provide interpretability through its cross-attention mechanism, specifically arguing that \"the areas with highest attention weights for the aTL-faces parcel primarily overlap with faces, while the attention weights for the mfs-words parcel overlap with words (line 537).\" However, Figure 8 appears to show attention weights that are merely high at the center of the image, rather than precisely aligning with the positions of faces and words. How can you more rigorously verify that the attention mechanism truly focuses on the selective object (face or word) rather than simply a central region?\n\n4. To more directly assess the object-specific attention effect of your cross-attention mechanism, consider presenting multi-cue images that contain both a face and a word within the same image. How would you then visualize and analyze the attention weights specifically for the aTL-faces and mfs-words parcels in this scenario? This could provide clearer evidence of selective attention.\n\n**Minor issues**\n\n1. Figure 4 captions: last (b) actually means (e)?\n\n2. line 539: “previous work using a similar encoder model [] “ -> Empty reference\n\n3. line 890: “The paper doe snot\" should be corrected to “does not”", "limitations": "This manuscript proposes a model for mapping visual categorical selectivity; however, the lack of experimental validation for the suggested 'superstimuli' inherently limits the plausibility of the model.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 2, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\nThis manuscript introduces an in-silico approach using an encoder-decoder transformer with a cross-attention mechanism, capable of aligning high-dimensional deep network features with cortical responses, thereby enabling the synthesis and selection of images that maximally activate different cortical parcels without controlled stimuli and linear encoding models.\n\n**Weaknesses**\n\nAlthough this research offers an interesting approach to identifying superstimuli, the lack of empirical validation of these superstimuli in new fMRI experiments inherently highlights a weakness.", "was_revised": true, "final_justification": "I raised a few concerns regarding the proposed methods, but the authors' rebuttal resolved them. In addition, the authors' additional experiments are helpful to trust the usefulness and significance of the mapping results. Thus, I raised my score accordingly.", "rebuttal": "We are thankful to Reviewer ywAe for the detailed and helpful review. We ran additional experiments to address the concerns and incorporate these additional results in our revision.\n\n> **Q1) Reproducing the selectivity of other known areas**\n\nWe are able to reproduce the known selectivity for all the major known areas for all the subjects. Optimal stimuli from NSD test, ImageNet, and BrainDIVE all appear similar to those reported in the literature. We will include in our revision examples like Figure 4 (verifying aTL-faces selectivity) for each aforementioned area to demonstrate place-, body-, and word-selectivity.\n\n> **Q2) Reproducing characteristics of the visual hierarchy (V1 to IT)**\n\nWe optimized 32 superstimuli (16 per hemisphere) that maximally activate parcels in V1, V2, and V4 using the BrainDIVE framework. We additionally include results on FFA as a representative IT-cortex ROI.\n\nGenerated superstimuli for early visual areas reproduce the classic coarse-to-fine hierarchy: V1 stimuli appear as cluttered scenes filled with dense, repetitive texture; V2 images add composite color patches and rudimentary objects; V4 stimuli reveal smoother, recognizable object forms; and FFA images almost exclusively depict close-up faces, often with clear emotional expressions. We will include these superstimuli in the revision.\n\nTo quantify hierarchical properties, we examined whether the spatial-frequency content of our stimuli mirrors classical physiological findings (e.g., [1,2]). We computed the radial average power spectrum for each image [3] and calculated the proportion of spectral power above 10%, 20%, and 30% of the maximum spatial frequency. At every threshold the high-frequency energy ratio decreases monotonically from V1 > V2 > V4 > FFA, indicating that the images that best drive higher-level areas (FFA) contain proportionally less fine-scale texture and relatively more coarse, low-frequency structure.\n\n| Threshold | ROI | High-frequency energy ratio |\n|-|-|-|\n| 0.1 | V1 | 0.006547 |\n| 0.1 | V2 | 0.004160 |\n| 0.1 | V4 | 0.002500 |\n| 0.1 | FFA| 0.001347 |\n| 0.2 | V1 | 0.001838 |\n| 0.2 | V2 | 0.000944 |\n| 0.2 | V4 | 0.000699 |\n| 0.2 | FFA| 0.000299 |\n| 0.3 | V1 | 0.000762 |\n| 0.3 | V2 | 0.000339 |\n| 0.3 | V4 | 0.000285 |\n| 0.3 | FFA| 0.000104 |\n\nNote: the natural image datasets (ImageNet, NSD) did not show as clear of a V1 > V2 > V4 > FFA ordering because their spectra cluster around natural photo statistics, masking early-visual preferences.\n\n> **Q3/4) Attention mechanism criticism and suggestions for improvement**\n\nWe acknowledge that the attention weights in our models tend to have a strong center bias, and suspect a few possible causes. First, the images that maximally activate a given parcel tend to have the object front and center. This is to be expected as there exist more voxels that represent centrally fixated objects. Second, the Schaefer parcellation is based on functional connectivity and does not necessarily respect categorical boundaries which can weaken the attention signal. We do agree the experiment you are suggesting would target the attention maps more directly, and we will explore them in future work.\n\nWe would like to note that the attention maps are only included in the supplementary and not mentioned in the core text. We believe it is one promising avenue for interpretability using in silico mapping methods, but not a core contribution of our study. We will include in the revision an additional interpretability metric, demonstrating with representational similarity analysis that learned transformer ROI parcel queries reproduce the resting state functional connectivity between those parcels. See our rebuttal to Q1 from Reviewer jfLo for more details.\n\n> **Q5) Typos in figure and text**\n\nThank you for pointing these out. In the Figure 4 caption, the last (b) should actually be (e). The empty reference should be to [4]. We will fix these in the revision.\n\n[1] David H. Hubel and Torsten N. Wiesel. Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. The Journal of Physiology, 160(1):106–154, January 1962. doi:10.1113/jphysiol.1962.sp006837. URL\n\n[2] R. Desimone and S. J. Schein. Visual properties of neurons in area V4 of the macaque: sensitivity to stimulus form. Journal of Neurophysiology, 57(3):835–868, March 1987. doi:10.1152/jn.1987.57.3.835. URL\n\n[3] A. van der Schaaf and J. H. van Hateren. Modelling the power spectra of natural images: Statistics and information. Vision Research, 36(17):2759–2770, September 1996. doi:10.1016/0042-6989(96)00002-8. URL\n\n[4] Hossein Adeli, Minni Sun, and Nikolaus Kriegeskorte. Transformer brain encoders explain human high‑level visual responses, 2025. URL"}, {"review_id": "yxYa1fqQlH", "reviewer": "Reviewer_kyMo", "summary": "This paper introduces a transformer-based encoder-decoder model ensemble to predict fMRI parcel vertex activity from image stimuli and use this as an in-silico model of the brain to study visual selectivity of cortical regions.  The introduced ensemble is composed of multiple frozen image encoder backbones and trainable decoders to predict fMRI responses.  The authors demonstrate the use of this ensemble model in assigning semantic meaning to parcels in and outside of the visual cortex (within single subjects and across multiple subjects) as a means to efficiently generate hypotheses about categorically selective regions of the brain.", "questions": "- Ensemble ablation: What is the variability in fMRI predictivity among the individual models of the ensemble?\n- Comparison to existing baselines: How much better does the proposed ensemble (and also individual models in the ensemble) predict fMRI activity compared to pre-trained vision (e.g., ResNet-50, DINO, etc.) models with learned affine transformations from intermediate layers?\n- Comparison with existing work: Where do you suggest this method stands in comparison to pre-trained models that have been aligned with cortical regions of the brain with affine transformations and diffusion-based methods (e.g., MindEye) that synthesize images based on in-silico fMRI activity (thereby offering a platform for analyzing selectivity of different brain region)? Any concrete examples of what the proposed method can reveal and what existing methods cannot would help clarify the contributions of this work.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "Strengths:\n- Practical Utility in Experimental Design:  The paper presents an intuitive framework for selecting brain parcels for future fMRI studies, particularly through its “Choosing parcels for future fMRI experimentation” section. This provides experimenters with a principled and cost-effective method to prioritize regions for follow-up studies based on encoder-derived selectivity metrics.\n- Mapping selectivity beyond classical visual areas:  The paper offers proof-of-concept results (proof-of-concept in the sense that in some cases selectivity hypotheses were generated by the model but not verified in-vivo) for identifying selectivity of regions outside of canonical visual areas. This type of analysis could offer insight into integration of visual information into regions of the brain that are not directly involved in visual processing.\n- Quantitative evaluation of hypotheses:  The authors go beyond qualitative image analysis and validate parcel selectivity hypotheses using semantic similarity metrics and ground-truth fMRI data.\n\nWeaknesses:\n- Positioning relative to prior work: While the paper introduces an ensemble transformer model, many prior works have also trained DNNs on the NSD dataset to predict brain responses and use these models to infer selectivity of arbitrary voxels.  It is unclear to me how this method substantially differs in outcome or capability beyond architectural novelty.\n- Lack of baseline comparisons: A more direct comparison between existing models that predict fMRI activity from image stimuli or ablation would help clarify the need for the proposed model ensemble (Does the proposed model predict fMRI activity more accurately than existing methods? Are the learnable parcel queries necessary compared to vanilla decoders?).  The authors suggest that previous works are limited by the fact that they frequently rely on the use of an affine transformation from model representations to predict fMRI activity, but do not seem to provide evidence that their proposed model offers more accurate predictions.\n- Method is limited to regions of the brain that are categorically and semantically selective:  While the model demonstrates semantic selectivity for higher-order concepts and object categories, it does not convincingly show whether it can uncover selectivity for lower-level visual features (e.g., texture, orientation, motion), which are critical components of the visual processing hierarchy.", "was_revised": true, "final_justification": "In follow-up responses with the authors, they have helped to justify, and provide quantitative results, for the unsupported claims in the paper that I had highlighted in my review.  Additionally, the authors helped to clarify their model design choice and why this is an improvement over comparable models used for similar applications.\n\nThe primary unresolved issue that remain include (1) limited comparison to existing baselines in terms of how well the models can explain variance in well known voxels (not the entire brain, but rather common areas such as V1, V2, V2, IT) and (2) evaluating these baselines for in-silico mapping on the same datasets to verify the improvements from the proposed approach.", "rebuttal": "We thank Reviewer kyMo for the helpful review. We address specific questions below.\n\n> **Q1) Positioning relative to prior work**\n\nIt is great that in silico mapping of the brain has received much attention in recent years, with many great papers that we build off of. Our method offers several key innovations:\n\n1) **Massive scale.** We apply in silico mapping on a massive scale with ImageNet and BrainDIVE, made uniquely possible with the transformer encoder architecture. We demonstrate that our pipeline can successfully test parcel selectivity beyond those concepts explicitly shown to the participant. To the best of our knowledge, no other study has tested image datasets on this scale nor demonstrated the capability to test concepts beyond the training set in silico.\n2) **Mapping of the whole brain.** Previous work has focused on areas in the visual cortex (mostly V1-V4, some limited higher-level areas). We expand semantic selectivity mapping to the whole brain beyond the visual area. We discover areas with novel complex semantic selectivity beyond the classic visual area.\n3) **In silico verification.** Our pipeline can verify selectivity hypotheses in silico by evaluating how well a label can predict ground-truth activation on a held-out set.\n4) **New fMRI experimental paradigm.** As both image datasets and encoding models improve, our encoder-agnostic pipeline offers a way to leverage these advances to accelerate and improve the accuracy of whole-brain mapping.\n\n> **Q2) Reproducing selectivity of low-level areas in the visual hierarchy**\n\nWe can indeed reproduce the selectivity to low-level visual features in the early visual processing hierarchy. We optimized 32 superstimuli (16 per hemisphere) that maximally activate parcels in V1, V2, and V4 using the BrainDIVE framework. FFA is included as a representative ROI in IT cortex.\n\nGenerated superstimuli for early visual areas reproduce the classic coarse-to-fine hierarchy: V1 stimuli appear as cluttered scenes filled with dense, repetitive texture; V2 images add composite color patches and rudimentary objects; V4 stimuli reveal smoother, recognizable object forms; and FFA images almost exclusively depict close-up faces, often with clear emotional expressions. We will include these superstimuli in the revision.\n\nTo quantify hierarchical properties, we examined whether the spatial-frequency content of our stimuli mirrors classical physiological findings (e.g., [1,2]). We computed the radial average power spectrum for each image [3] and calculated the proportion of spectral power above 10%, 20%, and 30% of the maximum spatial frequency. At every threshold the high-frequency energy ratio decreases monotonically from V1 > V2 > V4 > FFA, indicating that the images that best drive higher-level areas (FFA) contain proportionally less fine-scale texture and relatively more coarse, low-frequency structure.\n\n| Threshold | ROI | High-frequency energy ratio |\n|-----------|-----|----------------------------|\n| 0.1 | V1 | 0.006547 |\n| 0.1 | V2 | 0.004160 |\n| 0.1 | V4 | 0.002500 |\n| 0.1 | FFA| 0.001347 |\n| 0.2 | V1 | 0.001838 |\n| 0.2 | V2 | 0.000944 |\n| 0.2 | V4 | 0.000699 |\n| 0.2 | FFA| 0.000299 |\n| 0.3 | V1 | 0.000762 |\n| 0.3 | V2 | 0.000339 |\n| 0.3 | V4 | 0.000285 |\n| 0.3 | FFA| 0.000104 |\n\nNote: the natural image datasets (ImageNet, NSD) did not show as clear of a V1 > V2 > V4 > FFA ordering because their spectra cluster around natural photo statistics, masking early-visual preferences.\n\n> **Q3) Ensemble ablation**\n\n**Encoding accuracy, CLIP backbone**\n\n| Subject / Hemi | Layer -1 Run 1 | Layer -1 Run 2 | Layer -3 Run 1 | Layer -3 Run 2 | Layer -5 Run 1 | Layer -5 Run 2 | Layer -7 Run 1 | Layer -7 Run 2 |\n| -------------- | -------------- | -------------- | -------------- | -------------- | -------------- | -------------- | -------------- | -------------- |\n|S1 LH|0.346|0.341|0.348|0.348|0.331|0.338|0.353|0.350|\n|S1 RH|0.345|0.354|0.342|0.355|0.337|0.353|0.347|0.342|\n|S2 LH|0.384|0.372|0.368|0.371|0.377|0.374|0.371|0.375|\n|S2 RH|0.403|0.395|0.402|0.418|0.414|0.404|0.405|0.410|\n|S5 LH|0.344|0.345|0.352|0.349|0.351|0.345|0.344|0.346|\n|S5 RH|0.331|0.331|0.325|0.341|0.324|0.332|0.328|0.335|\n|S7 LH|0.322|0.327|0.330|0.332|0.328|0.330|0.320|0.341|\n|S7 RH|0.306|0.317|0.313|0.303|0.309|0.316|0.309|0.317|\n\n**Encoding accuracy, DINOv2 (ViT-B) backbone**\n\n| Subject / Hemi | Layer -1 Run 1 | Layer -1 Run 2 | Layer -3 Run 1 | Layer -3 Run 2 | Layer -5 Run 1 | Layer -5 Run 2 | Layer -7 Run 1 | Layer -7 Run 2 |\n| -------------- | -------------- | -------------- | -------------- | -------------- | -------------- | -------------- | -------------- | -------------- |\n|S1 LH|0.367|0.361|0.367|0.360|0.339|0.340|0.306|0.300|\n|S1 RH|0.364|0.360|0.363|0.369|0.354|0.348|0.302|0.303|\n|S2 LH|0.315|0.312|0.411|0.422|0.392|0.403|0.356|0.358|\n|S2 RH|0.353|0.351|0.455|0.450|0.435|0.438|0.388|0.390|\n|S5 LH|0.378|0.380|0.369|0.368|0.368|0.367|0.332|0.341|\n|S5 RH|0.370|0.365|0.361|0.360|0.345|0.341|0.313|0.313|\n|S7 LH|0.344|0.348|0.339|0.332|0.327|0.336|0.307|0.304|\n|S7 RH|0.333|0.340|0.324|0.320|0.311|0.309|0.297|0.292|\n\n> **Q4) Comparison to existing baselines**\n\nThe transformer-based encoding model does outperform existing models, including those that rely on an affine transformation from model representations. As the focus of the manuscript centers around the other parts of the pipeline (which we note is encoder agnostic), we did not include those comparisons here. See [4] for a comparison of a similar encoder to past approaches, showing that the transformer-based approach greatly outperforms other pre-trained vision models with affine transformations.\n\nFurthermore, encoders that rely on affine transformations from model representations are not feasible when scaled to make predictions for the whole brain. Building a linear encoding model for the whole brain with the same size feature maps would have 242B parameters.† The transformer-based model learns parcel queries to reduce feature size to 768, while still outperforming the full linear model. An experiment comparing the two isn't feasible to conduct but [4] compares the performance of these architectures on the visual area.\n\nGiven the recent interest and advances in encoding models, our contribution emphasizes that a sufficiently good encoder can generate \"superstimuli\" superior to those presented to the subject, and uncover complex semantic selectivity beyond the classic visual area.\n\n†31\\*31 (patch size) \\* 768 (dimension size) \\* 327684 (voxels) = 242B parameters\n\n> **Q5) Comparison with existing work**\n\nAs Reviewer kyMo pointed out, decoding models (including MindEye [5]) can be used to study selectivity by manipulating neural activity in different areas and examining the observed changes in the reconstructed images. However, those decoding models have only been trained for the visual areas because they require learning very parameter-intensive mappings from the voxels to the intermediate representations needed as inputs to the diffusion models. Due to this limitation, to our knowledge, they have not yet been applied to decoding from whole-brain fMRI activity as they are not suitable for the type of experiments we are performing. Future more lightweight decoding models that can generalize to the whole brain would certainly be good baseline models for our approach. We will add a discussion of these models to the revision. \n\n[1] David H. Hubel and Torsten N. Wiesel. Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. The Journal of Physiology, 160(1):106–154, January 1962. doi:10.1113/jphysiol.1962.sp006837. URL\n\n[2] R. Desimone and S. J. Schein. Visual properties of neurons in area V4 of the macaque: sensitivity to stimulus form. Journal of Neurophysiology, 57(3):835–868, March 1987. doi:10.1152/jn.1987.57.3.835. URL\n\n[3] A. van der Schaaf and J. H. van Hateren. Modelling the power spectra of natural images: Statistics and information. Vision Research, 36(17):2759–2770, September 1996. doi:10.1016/0042-6989(96)00002-8. URL\n\n[4] Hossein Adeli, Minni Sun, and Nikolaus Kriegeskorte. Transformer brain encoders explain human high‑level visual responses, 2025.\n\n[5] Paul S. Scotti, Atmadeep Banerjee, Jimmie Goode, Stepan Shabalin, Alex Nguyen, Ethan Cohen, Aidan J. Dempster, Nathalie Verlinde, Elad Yundler, David Weisberg, Kenneth A. Norman, and Tanishq Mathew Abraham. Reconstructing the Mind’s Eye: fMRI-to-Image with Contrastive Learning and Diffusion Priors, 2023. URL"}]

Author rebuttal:

Meta-review: This paper proposes a method for mapping the selectivity to visual stimuli of cortical regions across the whole human brain. The paper is based on a digital twin trained to predict fMRI responses to visual stimuli from the Natural Scenes Dataset. The digital twin allows for large in-silico experiments, searching for images that are expected to strongly activate an area in a large database, or to use generative models to create synthetic superstimuli. The method can recover the selectivity of known areas as well as discover selectivity for novel concepts.

The reviewers appreciated the paper as a significant technical advance (especially after clarification about its relationship with previous literature on stimulus synthesis for rodents and NHPs and on fMRI response prediction using DNNs trained on NSD) and were satisfied with the degree of validation. The author-reviewer discussion generated a substantial amount of new analyses and results. Of particular interest was the replication of the selectivity of additional areas that are not categorically and semantically selective, such as the low-level areas in the visual hierarchy.

Overall, the paper will be of interest to the NeurIPS community and I recommend acceptance.
Decision: Accept (poster)