Title: In Silico Mapping of Visual Categorical Selectivity Across the Whole Brain
Venue: NeurIPS 2025
Accepted: yes
Keywords: neural encoding

brain mapping

transformers

generative diffusion model
Abstract: A fine-grained account of functional selectivity in the cortex is essential for understanding how visual information is processed and represented in the brain. Classical studies using designed experiments have identified multiple category-selective regions; however, these approaches rely on preconceived hypotheses about categories. Subsequent data-driven discovery methods have sought to address this limitation but are often limited by simple, typically linear encoding models. We propose an in silico approach for data-driven discovery of novel category-selectivity hypotheses based on an encoder–decoder transformer model. The architecture incorporates a brain-region to image-feature cross-attention mechanism, enabling nonlinear mappings between high-dimensional deep network features and semantic patterns encoded in the brain activity. We further introduce a method to characterize the selectivity of individual parcels by leveraging diffusion-based image generative models and large-scale datasets to synthesize and select images that maximally activate each parcel. Our approach reveals regions with complex, compositional selectivity involving diverse semantic concepts, which we validate in silico both within and across subjects. Using a brain encoder as a “digital twin” offers a powerful, data-driven framework for generating and testing hypotheses about visual selectivity in the human brain—hypotheses that can guide future fMRI experiments. Our code is available at: https://kriegeskorte-lab.github.io/in-silico-mapping-web/.

In Silico Mapping of Visual Categorical Selectivity Across the Whole Brain

Abstract

A fine-grained account of functional selectivity in the cortex is essential for understanding how visual information is processed and represented in the brain. Classical studies using designed experiments have identified multiple category-selective regions; however, these approaches rely on preconceived hypotheses about categories. Subsequent data-driven discovery methods have sought to address this limitation but are often limited by simple, typically linear encoding models. We propose an in silico approach for data-driven discovery of novel category-selectivity hypotheses based on an encoder-decoder transformer model. The architecture incorporates a brain-region to image-feature cross-attention mechanism, enabling nonlinear mappings between high-dimensional deep network features and semantic patterns encoded in the brain activity. We further introduce a method to characterize the selectivity of individual parcels by leveraging diffusion-based image generative models and large-scale datasets to synthesize and select images that maximally activate each parcel. Our approach reveals regions with complex, compositional selectivity involving diverse semantic concepts, which we validate in silico both within and across subjects. Using a brain encoder as a 'digital twin' offers a powerful, data-driven framework for generating and testing hypotheses about visual selectivity in the human brain-hypotheses that can guide future fMRI experiments. Our code is available at: https://kriegeskorte-lab.github.io/in-silico-mapping/.

1 Introduction

Over the past few decades, researchers have extensively studied the visual hierarchy in the brain, from early cortical areas that encode low-level features to higher-level regions that represent categorical information. Neuroimaging experiments, especially studies using functional magnetic resonance imaging (fMRI), have revealed specialized cortical regions for faces, places, words, bodies, and food [48, 35, 5, 63, 15, 8, 21, 20, 61, 65, 34, 38, 69, 54]. However, visual perception is more nuanced than this short list of categories. It remains unknown what additional visual concepts have

dedicated regions that enable humans to make sense of the complex world. Common mapping methods depend on experimenter-curated concepts, and empirically-driven alternatives require more data and expensive fMRI experiments. For example, one could present a subject with a large set of images and then label the selectivity of each cortical parcel by the images that elicit the strongest mean parcel response. As the stimulus set grows, however, the cost of data acquisition (operating the scanner, paying subjects, bonuses)-at least with current fMRI technology and experimental paradigms-may prove prohibitive. We propose methods that rely on the state-of-the-art encoding models to generalize to concepts beyond the stimuli for which fMRI responses have been measured.

(a)

Brain

(q)

Parcel

responses

vertices

↑

Image

Linear

Linear

encoder

装

Feedforward

Feedforward

Attention

scores

Self Attention

CrossAttention

Scaled dot

product

?

X N

K

Q

Image

Positional

Parcel

Image

Positional Parcel

patches

encoding

queries

tokens

encoding

query

(c)

Subject specific

(d)

Shared across subjects

Skateboarding parcel

Child eating parcel

Dynamic bodyparcel

Tool parcel

Our encoding model leverages recent advances in AI and large-scale neural datasets to serve as a 'digital twin,' upon which we perform extensive in silico experimentation [27, 29] to generate hypotheses of complex categorical selectivity beyond the visual cortex. The encoding model enables us to predict the neural activity of image sets far larger and more diverse than the limited number of images shown to the participant during fMRI, effectively expanding the search space for optimal visual stimuli. In addition, the encoding model is fully observable and differentiable, enabling interpretability queries using attention maps and gradient-based analyses, such as diffusion-based models that use gradients to find stimuli that elicit high activity [46].

Through experimentation with large image datasets on our model, we generated hypotheses for the selectivity of many parcels outside the visual area, as well as optimal sets of images that maximally activate the parcels. These hypotheses can be tested in targeted future fMRI experiments by showing only the sets of optimal stimuli, accelerating data collection and experiment iteration, and lowering

the cost of data acquisition. We demonstrate that our pipeline using brain encoders can test concepts that were not explicitly shown to the subject in the scanner, effectively enriching the diversity and size of the fMRI training set.

Key contributions of our work:

Massive scale: applying in silico mapping on millions of images (ImageNet, BrainDIVE) with a transformer-based brain encoder, enabling discovery of parcel selectivity for concepts never shown in training. To the best of our knowledge, no other study has been done on this scale.

Mapping of the whole brain: expanding beyond visual cortex and revealing human-specific semantic selectivity.

In silico verification: our pipeline verifies selectivity hypotheses in silico with rigorous tests that evaluate how well a label can predict ground-truth activation on a held-out set within and across subjects.

New fMRI experimental paradigm: as datasets grow and encoding models improve, our pipeline offers a way to leverage these advances to accelerate and improve the accuracy of whole-brain mapping.

2 Related Work

Semantic mapping. Our work builds upon a growing body of computational modeling and machine learning research that investigates how semantic information is represented in the higher visual cortex [42]. Some approaches leverage large image datasets to build decoders [19, 22, 66, 36, 62, 74, 14, 44] or models for generating optimal stimuli [13, 46, 45, 28, 58, 47], while others use cross-domain (e.g. vision-language) mapping [33, 32, 37, 9, 50, 51, 55, 64, 30]. These studies face a key challenge of dataset size, since the collection of neural data is often expensive and time-consuming. Our work seeks to address this constraint by using encoders trained on large-scale datasets to perform in silico mapping. This allows us to expand the set of concepts that can be probed, beyond those stimuli shown to the subjects.

Brain encoding models. Highlighting its importance, several community-driven efforts have sought to benchmark models predicting brain responses [60, 68, 26]. With the release of increasingly large neural datasets each year, researchers have introduced novel architectures and methodologies to improve the accuracy of brain encoding model, including leveraging multiple datasets and pretrained networks [2, 43, 73, 11, 72, 57, 24, 41, 67, 7]. While our paper uses a cutting-edge encoding model [3], our pipeline is ultimately encoder-agnostic, and can use any encoder that is image-computable. As researchers build better brain encoders, we expect that the space of hypotheses our pipeline could generate and their accuracy will only grow.

Brain-optimized stimuli. Previous studies have introduced encoding model-based stimulus selection and empirically validated the superstimuli in non-human primates and mice [10, 71, 56, 23]. In this work, we extend this general approach of stimulus optimization for studying neural populations to the fMRI domain, revealing high-level human-specific selectivity beyond the visual cortex.

3 Methods

Our goal is to map the visual selectivity of parcels across the whole brain. First, we train a brain encoder to predict fMRI responses from natural scene images. Then we select visually responsive and robust parcels for experimentation to determine categorical selectivity.

3.1 Parcellation Strategy

We partitioned the 327,684 cortical vertices across the whole brain into 1,000 functional parcels using the Schaefer resting-state functional connectivity parcellation (see Figure 2) [59].

3.2 Brain Encoder Architecture

Extending the work of Adeli et al. [2], our brain encoder predicts vertex-wise fMRI activity across the whole brain from an input image (see Figure 1a). An image encoder backbone (see Section 4.2) first extracts patch embeddings from the image. A transformer decoder uses parcel-specific, learnable queries to attend to relevant patch embeddings via cross-attention. The decoder consists of a single transformer layer with cross-attention followed by a feedforward projection (see Figure 1b for crossattention). Each output decoder token is linearly mapped to predict the fMRI responses for vertices in the corresponding parcel, and predictions from all output tokens are aggregated to obtain a wholebrain prediction. Parcel queries, the transformer decoder, and linear mappings are optimized using Adam [39] to minimize the mean squared error between the predicted and actual fMRI responses. All other layers, including the backbone, are frozen. Separate models are trained for each subject.

To improve prediction accuracy, we ensemble multiple instances of the brain encoder. For each subject, we trained two random seeds with features from four different DINOv2 backbone layers (the 0th, 2nd, 4th, and 6th layers from the last). To predict a vertex, we take the weighted average across model predictions, scaled by softmax weights from validation set accuracy for each model on that vertex.

3.3 Parcel selection process for further experimentation

Not all regions in the brain are visually responsive, so we selected parcels for further experimentation that satisfy three criteria: (1) Location: Fewer than 10% of parcel vertices overlap with the labeled area, since we are most interested in parcels beyond the visual cortex. (2) Visual responsiveness: The average noise ceiling must be in the top 25% of parcels that satisfy (1). (3) Model prediction accuracy: The average prediction accuracy must be in the top 25% of parcels that satisfy (1).

Since the data quality varies slightly across subjects, we used percentiles rather than numerical cutoffs. The selection process for parcels in subject 1's left hemisphere is shown in Figure S1 in Appendix A.3. Cutoffs are determined separately for each hemisphere to maintain comparable parcel counts.

Among the 500 Schaefer parcels in each hemisphere for every subject, 409 ± 6 parcels satisfy condition (1). After filtering out parcels with low mean noise ceiling or low mean model prediction accuracy, 179 ± 12 parcels per subject are chosen for further experimentation.

3.4 Superstimulus Generation Process

We choose images that maximally activate (mean z-scored beta values) a parcel of interest using three different methods:

Natural Scenes Dataset (NSD) Ground Truth Images: Images from the held-out NSD [4] test split, ranked based on ground truth data.

Diffusion-generated superstimuli: BrainDIVE [46] uses a generative backbone guided by gradients from a brain encoder to generate images that can maximally activate specified brain parcels. We generated 400 images per parcel and reranked them with the brain encoder.

Encoder-selected ImageNet superstimuli: ImageNet [17] images that maximally activate the parcel, according to the encoder.

4 Experiments

4.1 Setup

We used the NSD [4], the largest fMRI dataset to date, with 7T fMRI responses from 8 subjects who each viewed up to 10,000 distinct natural scenes. Each image is presented up to three times, and our model is trained on the neural response averaged over the presentations. We report results for subjects who completed all NSD scan sessions (1, 2, 5, and 7), though we observed comparable results in all subjects. fMRI responses were preprocessed according to [4]. The resulting beta estimates were centered to zero mean and scaled to unit variance before training and experiments. ROI labels were obtained from NSD. V1-hV4 ROIs are derived from a pRF experiment; body-, face-, place-, word-selective ROIs are derived from a fLoc experiment [4].

4.2 Brain encoder results

Model prediction accuracy. Figure 3 shows the performance of the ensemble model with DINOv2 (ViT-B) backbone and the transformer cross-attention mapping function for subject 1 projected onto the cortical surface using Pycortex [25]. As expected, the model performs well on predicting the activity in the visual cortex (the area in the center of the flatmap), but also on several regions beyond the typical visual pathways.

Model comparison. We evaluate several encoding models with different backbones and mapping functions to find the most suitable model for whole-brain voxel-wise prediction. The Algonauts 2023 challenge [26] leaderboard showed that transformer-based backbones generally outperformed other model families such as convolutional networks (CNNs) at predicting fMRI activity. Therefore, we focused on evaluating features from several transformer backbones paired with either transformer attention-based mapping to neural data [2] or a parameter-matched model that linearly maps the CLS token to vertex values. For each architecture, we used an ensemble of 16 models for each subject (4 feature backbone layers x 2 hemispheres x 2 random seeds). We compared the encoding accuracy, which is Pearson's correlation on the held-out set corrected for noise ceilings (see [4] Methods, Noise ceiling estimation).

Across the subjects that completed all NSD scan sessions (Table 1), the transformer models with the DINOv2 backbone [53] outperform the other non-DINOv2 backbones. The two DINOv2 backbones perform similarly, which is consistent with past work [1] that showed diminishing or worse performance for DINOv2 models larger than the base (ViT-B). The transformer-based encoder

significantly outperformed the linear baseline, leveraging the attention mechanism to flexibly route information [2, 6, 56]. Figure 4a plots the difference in prediction accuracy (Pearson's correlation) between the two.

Figure 4b plots the difference in prediction accuracy between DINOv2 and CLIP vision. While DINOv2 performs far better in the visual cortex, they perform similarly in many regions outside the visual area. For the rest of the paper, we performed our experiments on the brain encoder with DINOv2 (ViT-B) + Transformer, since it offered the best overall speed-performance tradeoff.

(a)

(b)

Pearson correlation

0.50.0

0.5

Pearson correlation

-0.20.0

Z'0

4.3 Sanity check on known regions

We first validate our paradigm by replicating the well-documented category selectivity of ventral pathway categorical areas. We show the results for a sample parcel from the labeled area, one which significantly overlaps with aTL-faces (47.6% of the vertices in the parcel overlap with aTL-faces). As shown in Figure 5, the images maximally activating the parcel overlapping with aTL-faces prominently feature faces, which agrees with previous work on the selectivity of this area [63]. In Appendix A.2, we reproduce the selectivity of body-, place-, and word-selective areas. In Appendix A.4, we show that stimuli generated by BrainDIVE reproduce the fine-to-coarse visual hierarchy progressing from V1-V4 to FFA.

4.4 Extension to unlabeled areas

We now extend our analyses to parcels beyond the visual cortex. We show images selected by the encoder to maximally activate each parcel for qualitative evaluation, then verify that our labels indeed explain parcel activations using formal statistical tests.

Labeling unlabeled parcels. We apply the same image-selection method to generate selectivity hypotheses for unlabeled parcels. Figure 1c displays ImageNet images predicted by the encoder to maximally activate two unlabeled subject-specific parcels: one appears to depict skateboarding, the other a child eating food. Full results, including parcel location and the corresponding NSD and BrainDIVE images, are shown in Figures S5 and S6 (Appendix A.7). Many parcels with selectivity that appeared to be consistent and complex were found across the whole brain, suggesting that there exist regions in the brain that respond to more complex concepts than the basic categories labeled in visual cortex.

Evaluating our labels. Our sanity checks demonstrated that our brain encoder retrieves images that align with the category selectivity of the labeled parcels. However, since the parcels we are interested in are outside the visual area and therefore unlabeled, we would like to quantitatively show that the images selected by our encoder faithfully reflect underlying neural selectivity.

We conducted two tests against the NSD test set-the only held-out fMRI data not used during training. If the selected images (hereafter 'labels') genuinely reflect parcel selectivity, other images with greater semantic similarity to the label should result in higher parcel activation. Prior studies have shown that categorical selectivity is not binary but graded across diverse stimuli [12, 49]. For

(a)

B

(e)

ImagenetvsGen.ImagesActivationDist

70000

Imagenet

60000

ImagenetClLower

50000

Imagenet CI Upper

GenCI Lower

GenCIUpper

R

30000

20000

10000

-0.3-0.2-0.10.0

0.20.3

0.4

Average predicted activation magnitude (z-scored)

(q)

NSD

(c)

Generated

(d)

Imagenet

example, fMRI responses to 1,705 object and action categories revealed that brain-response similarity strongly correlates with semantic similarity [33]. Accordingly, we expect a parcel's mean activity to scale with how semantically close an image is to that parcel's preferred concept, as defined by our encoder's maximally activating examples.

In each test, we compared two predictors: (1) labels derived from our encoder, and (2) a baseline that forms its hypothesis from each parcel's most activating NSD-training images, selected using their measured responses. This baseline mirrors conventional fMRI studies that rely solely on stimuli shown in the scanner. If our encoder performs better than the baseline training set, it shows we can use brain encoders to discover novel concepts not explicitly shown to the subject.

For Test 1, we asked whether semantic similarity to a label predicted the activation rank order of NSD test images better than chance. Our pipeline represents a label as the mean CLIP embedding of the top 32 ImageNet or BrainDIVE images (both shown separately); the baseline uses the top 32 NSD-training images. Appendix A.10 details the procedure. Table 2 reports the number of parcels for which each label significantly outperforms a random ranking. To ensure that the results reflect an expansion of the stimulus space and not merely dense sampling near peak activation, Appendix A.16 reports the same results with a varying number of top images used to create the label. Each parcel whose model-derived label outperforms the null is deemed successfully labeled. Across tests, the encoder-selected stimuli generally outperformed the baseline, labeling a greater number of parcels.

For Test 2, we compare Spearman's rank correlations between the ground-truth activation ordering and each model's predicted ordering, quantifying which model's selectivity hypothesis best explains parcel activations. Table 3 summarizes these coefficients across all parcels. A head-to-head comparison between our encoder and the baseline appears in Table 8 (Appendix A.13).

Notes. INet = ImageNet. BD = BrainDIVE.

The rank ordering derived from the encoder-selected stimuli generally outperforms the baseline ordering based on the NSD test set. To contextualize the rank correlation magnitude, we report in Appendix A.15 the results for known parcels in the visual area as a benchmark for what constitutes meaningful selectivity. The average rank correlation is in the range of areas like PPA and RSC-both widely studied and accepted in the literature. Because these encoder-driven selectivity hypotheses better explain activation patterns in the NSD test images, our results indicate that the brain-encoder pipeline can generate finer-grained categorical hypotheses than those afforded by the stimuli actually shown to the subject in the scanner.

Choosing parcels for future fMRI experimentation. For an fMRI study aimed at uncovering the selectivity of parcels outside visual cortex, an experimenter may be interested in parcels whose activity can be well explained by a semantic label. To prioritize such parcels, we define a metric that ranks the quality of the hypotheses generated by our pipeline. In Figure 6, we plot the proportion of top 32 maximally-activating images successfully retrieved against the number of images retrieved by the pipeline. The top five parcels per subject (ranked by area under the curve) are highlighted.

2 5

2 6

2 7

2 8

2 9

Number of Retrieved Images

0.0

0.2

0.4

0.6

0.8

1.0

Subject 1

LH 211

RH 373

LH 238

RH 456

LH 496

95% CI

2 5

2 6

2 7

2 8

2 9

Number of Retrieved Images

0.0

0.2

0.4

0.6

0.8

1.0

Subject 2

LH 480

RH 391

LH 500

LH 496

LH 365

95% CI

2 5

2 6

2 7

2 8

2 9

Number of Retrieved Images

0.0

0.2

0.4

0.6

0.8

1.0

Subject 5

RH 476

LH 239

LH 367

RH 391

LH 167

95% CI

2 5

2 6

2 7

2 8

2 9

Number of Retrieved Images

0.0

0.2

0.4

0.6

0.8

1.0

Subject 7

RH 474

RH 477

LH 496

LH 497

RH 317

95% CI

NSD test Imagenet Retrieval Rate

From an experimenter's perspective, the parcels shown in Figure 6 may be promising targets for follow-up fMRI studies, since semantic labels seem to capture the parcel selectivity exceptionally well. Our pipeline identifies images whose semantic representations appear to maximally activate these parcels-at least insofar as this can be evaluated with the modest NSD test set.

4.5 Mapping Cross-subject Selectivity

We now explore the selectivity of a parcel outside visual cortex (Figure 7a) by examining ImageNet images that the encoder predicts will maximally activate it. From the top 9 images per subject, we display 3 from each participant who completed all NSD scanning sessions. These maximallyactivating images lie substantially outside the parcel's activation distribution for the remainder of

ImageNet (Figure 7c). The selected images consistently depict hands manipulating tools-such as writing instruments or cooking utensils (see full collages in Appendix A.9). When asked to identify a common theme among the top 25 ImageNet images, ChatGPT likewise highlights hands with objects. Recent work has reported tool-use representations in nearby cortical regions [16].

0

Average predicted activation magnitude (z-scored)

(a)

(q)

"...utensils in

action, often

S1

in practical,

hands-on

settings.."

"...hands

actively

S2

engaging in a

step of making

food.."

"...objects or

body parts being

(c)

S5

used as writing

instruments or

Subject 1 All Imagenet vs Top-9 Activation

tools..."

100000

All Imagenet

80000

AllImagenet CI Lower

All ImagenetCI Upper

Frequency

60000

Imagenet Top-9 CI Lower

"...action

40000

S7

involving tools

or sports

20000

equipment..."

-0.6-0.4-0.20.00.20.40.60.8

Evaluating our labels. We qualitatively showed that our pipeline can uncover a parcel exhibiting consistent semantic selectivity across subjects. We now seek to quantitatively verify these hypotheses on ground-truth data. We ran statistical test similar to that in Section 4.4, with two key differences because we are evaluating selectivity across subjects.

Combining across subjects: For each parcel, we form the optimal image set by combining the top-32 maximally-activating images chosen by encoder models trained on the other three subjects.

Retrieval set: Because each subject's NSD training set contains distinct images, the ranking retrieval is performed on that subject's own NSD training set.

We again compare the selectivity hypotheses generated by our brain-encoder pipeline with the NSDtraining-set baseline (described in the within-subject statistical tests). Table 4 reports the results of this modified test (cf. Table 2); additional details are provided in Appendix A.11.

Comparing the number of parcels for which each hypothesized label ranks NSD-training images better than chance, our pipeline generally outperforms the NSD-train baseline. These findings indicate that for parcels with high visual responsiveness shared across subjects, our pipeline can generate better hypotheses that describe the shared selectivity.

Table 5 presents results from the same statistical test used in Table 3, reporting the average Spearman rank correlations for each model across parcels. A head-to-head comparison appears in Table 9 (Appendix A.13), and Figure S8 (Appendix A.14) highlights the most promising cross-subject parcels for future fMRI experiments.

Notes. INet = ImageNet. BD = BrainDIVE.

5 Discussion

Leveraging recent advances in AI and the availability of large-scale datasets, we introduce a datadriven paradigm for discovering parcel selectivity beyond visual cortex, paving the way toward systematic whole-brain labeling of higher-order visual representations. In particular, our transformerbased encoder, with its cross-attention mechanism and nonlinear mappings, routes visual information more effectively for parcels in and outside classical visual areas.

Limitations. Although we predict images that should maximally activate a parcel, we have not yet validated these 'superstimuli' in new fMRI experiments. On a held-out NSD split we show that our semantic labels predict parcel activations well, but whether the synthesized superstimuli elicit even stronger responses remains unknown. Moreover, because the encoder was trained solely on NSD [4], it may inherit dataset biases. For example, one parcel in subject 2's left hemisphere appears zebra-selective (Figure S10, Appendix A.19), however, this could be due to the over-representation of giraffe and zebra images in the dataset. We did not quantify how such biases affect our results; future work should train on multiple datasets to address this concern.

More fundamentally, category selectivity is limited as a theoretical construct for understanding brain computation. We hope our approach will help the field go beyond the technical limitations of prior work and help reveal the fundamental limits of understanding brain computation through the lens of category selectivity.

Optimizing future fMRI experiments. Our framework paves the way for future fMRI studies that mitigate the effect of small datasets and experimenter bias in image selection by demonstrating the promise of in silico mapping to superstimuli. These superstimuli can then be tested in follow-up scans to verify the selectivity of newly discovered areas. By letting the encoder drive hypothesis generation, researchers can discover optimal stimuli empirically-even for concepts never presented in the scanner-maximizing data-collection efficiency. Because our encoder architecture is modalityagnostic, the same semantic-mapping approach could be extended to multi-modal backbones, enabling superstimuli generation across sensory domains.

Acknowledgments

Research reported in this publication was supported in part by the National Institute of Neurological Disorders and Stroke of the National Institutes of Health under award numbers 1RF1NS128897 and 4R01NS128897. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

References

Hossein Adeli, Seoyoung Ahn, Nikolaus Kriegeskorte, and Gregory Zelinsky. Affinity-based attention in self-supervised transformers predicts dynamics of object grouping in humans, 2023. URL https://arxiv.org/abs/2306.00294 .

Hossein Adeli, Sun Minni, and Nikolaus Kriegeskorte. Predicting brain activity using Transformers, August 2023. URL http://biorxiv.org/lookup/doi/10.1101/2023.08.02. 551743 .

Hossein Adeli, Minni Sun, and Nikolaus Kriegeskorte. Transformer brain encoders explain human high-level visual responses, May 2025. URL http://arxiv.org/abs/2505.17329 . arXiv:2505.17329 [q-bio].

Emily J. Allen, Ghislain St-Yves, Yihan Wu, Jesse L. Breedlove, Jacob S. Prince, Logan T. Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, J. Benjamin Hutchinson, Thomas Naselaris, and Kendrick Kay. A massive 7T fMRI dataset to bridge cognitive neuroscience and artificial intelligence. Nature Neuroscience , 25(1):116-126, January 2022. ISSN 1097-6256, 1546-1726. doi: 10.1038/s41593-021-00962-x. URL https://www.nature.com/articles/s41593-021-00962-x .

Truett Allison, Gregory McCarthy, Anna Nobre, Aina Puce, and Aysenil Belger. Human extrastriate visual cortex and the perception of faces, words, numbers, and colors. Cerebral cortex , 4(5):544-554, 1994.

Mehdi Azabou, Vinam Arora, Venkataramana Ganesh, Ximeng Mao, Santosh Nachimuthu, Michael Mendelson, Blake Richards, Matthew Perich, Guillaume Lajoie, and Eva L. Dyer. A unified, scalable framework for neural population decoding. In Thirty-seventh Conference on Neural Information Processing Systems , 2023.

Mehdi Azabou, Vinam Arora, Venkataramana Ganesh, Ximeng Mao, Santosh Nachimuthu, Michael J. Mendelson, Blake Richards, Matthew G. Perich, Guillaume Lajoie, and Eva L. Dyer. A unified, scalable framework for neural population decoding, 2023. URL https: //arxiv.org/abs/2310.16046 .

Chris I. Baker, Jia Liu, Lawrence L. Wald, Kenneth K. Kwong, Thomas Benner, and Nancy Kanwisher. Visual word processing and experiential origins of functional selectivity in human extrastriate cortex. Proceedings of the National Academy of Sciences of the United States of America , 104(21):9087-9092, May 2007. ISSN 0027-8424 1091-6490. doi: 10.1073/pnas. 0703300104. Place: United States.

Alfonso Barrós-Loscertales, Julio González, Friedemann Pulvermüller, Noelia Ventura-Campos, Juan Carlos Bustamante, Víctor Costumero, María Antonia Parcet, and César Ávila. Reading salt activates gustatory brain regions: fMRI evidence for semantic grounding in a novel sensory modality. Cerebral cortex (New York, N.Y. : 1991) , 22(11):2554-2563, November 2012. ISSN 1460-2199 1047-3211. doi: 10.1093/cercor/bhr324. Place: United States.

Pouya Bashivan, Kohitij Kar, and James J DiCarlo. Neural population control via deep image synthesis. Science , 364(6439):eaav9436, 2019.

Roman Beliy, Navve Wasserman, Amit Zalcher, and Michal Irani. The wisdom of a crowd of brains: A universal brain encoder, 2025. URL https://arxiv.org/abs/2406.12179 .

Edwin J. Burns, Taylor Arnold, and Cindy M. Bukach. P-curving the fusiform face area: Meta-analyses support the expertise hypothesis. Neuroscience & Biobehavioral Reviews , 104: 209-221, 2019. ISSN 0149-7634. doi: https://doi.org/10.1016/j.neubiorev.2019.07.003. URL https://www.sciencedirect.com/science/article/pii/S014976341830798X .

Diego García Cerdas, Christina Sartzetaki, Magnus Petersen, Gemma Roig, Pascal Mettes, and Iris Groen. Brainactiv: Identifying visuo-semantic properties driving cortical selectivity using diffusion-based image manipulation. bioRxiv , 2024. doi: 10.1101/2024.10.29.620889. URL https://www.biorxiv.org/content/early/2024/10/31/2024.10.29.620889 .

Zijiao Chen, Jiaxin Qing, Tiange Xiang, Wan Lin Yue, and Juan Helen Zhou. Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding. arXiv preprint arXiv:2211.06956 , 1(2):4, 2022.

L. Cohen, S. Dehaene, L. Naccache, S. Lehéricy, G. Dehaene-Lambertz, M. A. Hénaff, and F. Michel. The visual word form area: spatial and temporal characterization of an initial stage of reading in normal subjects and posterior split-brain patients. Brain : a journal of neurology , 123 ( Pt 2):291-307, February 2000. ISSN 0006-8950. doi: 10.1093/brain/123.2.291. Place: England.

Davide Cortinovis, Marius V. Peelen, and Stefania Bracci. Tool Representations in Human Visual Cortex. Journal of Cognitive Neuroscience , 37(3):515-531, March 2025. ISSN 0898929X. doi: 10.1162/jocn_a_02281. URL https://doi.org/10.1162/jocn_a_02281 .

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A largescale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 248-255, Miami, FL, June 2009. IEEE. ISBN 978-1-4244-3992-8. doi: 10. 1109/CVPR.2009.5206848. URL https://ieeexplore.ieee.org/document/5206848/ .

R. Desimone and S. J. Schein. Visual properties of neurons in area V4 of the macaque: sensitivity to stimulus form. Journal of neurophysiology , 57(3):835-868, March 1987. ISSN 0022-3077. doi: 10.1152/jn.1987.57.3.835. Place: United States.

Adrien Doerig, Tim C Kietzmann, Emily Allen, Yihan Wu, Thomas Naselaris, Kendrick Kay, and Ian Charest. Semantic scene descriptions as an objective of human vision. arXiv preprint arXiv:2209.11737 , 2022.

Paul E Downing, Yuhong Jiang, Miles Shuman, and Nancy Kanwisher. A cortical area selective for visual processing of the human body. Science , 293(5539):2470-2473, 2001.

Russell Epstein and Nancy Kanwisher. A cortical representation of the local visual environment. Nature , 392(6676):598-601, 1998.

Matteo Ferrante, Furkan Ozcelik, Tommaso Boccato, Rufin VanRullen, and Nicola Toschi. Brain captioning: Decoding human brain activity into images and text. arXiv preprint arXiv:2305.11560 , 2023.

Katrin Franke, Nikos Karantzas, Konstantin Willeke, Maria Diamantaki, Kandan Ramakrishnan, Pavithra Elumalai, Kelli Restivo, Paul Fahey, Cate Nealley, Tori Shinn, et al. Dual-feature selectivity enables bidirectional coding in visual cortical neurons. bioRxiv , pages 2025-07, 2025.

Maëlle Freteault, Maximilien Le Clei, Loic Tetrel, Lune Bellec, and Nicolas Farrugia. Alignment of auditory artificial networks with massive individual fmri brain data leads to generalisable improvements in brain encoding and downstream tasks. Imaging Neuroscience , 3, 04 2025. ISSN 2837-6056. doi: 10.1162/imag\_a\_00525. URL https://doi.org/10.1162/imag_ a_00525 .

James S. Gao, Alexander G. Huth, Mark D. Lescroart, and Jack L. Gallant. Pycortex: an interactive surface visualizer for fMRI. Frontiers in Neuroinformatics , 9, September 2015. ISSN 1662-5196. doi: 10.3389/fninf.2015.00023. URL http://journal.frontiersin. org/Article/10.3389/fninf.2015.00023/abstract .

Alessandro T Gifford, Benjamin Lahner, Sari Saba-Sadiya, Martina G Vilas, Alex Lascelles, Aude Oliva, Kendrick Kay, Gemma Roig, and Radoslaw M Cichy. The algonauts project 2023 challenge: How the human brain makes sense of natural scenes. arXiv preprint arXiv:2301.03198 , 2023.

Alessandro T. Gifford, Maya A. Jastrz˛ ebowska, Johannes J. D. Singer, and Radoslaw M. Cichy. In silico discovery of representational relationships across visual cortex, 2025. URL https://arxiv.org/abs/2411.10872 .

Zijin Gu, Keith Wakefield Jamison, Meenakshi Khosla, Emily J. Allen, Yihan Wu, Ghislain St-Yves, Thomas Naselaris, Kendrick Kay, Mert R. Sabuncu, and Amy Kuceyeski. NeuroGen: Activation optimized image synthesis for discovery neuroscience. NeuroImage , 247:118812, February 2022. ISSN 10538119. doi: 10.1016/j.neuroimage.2021.118812. URL https: //linkinghub.elsevier.com/retrieve/pii/S1053811921010831 .

Umut Güçlü and Marcel A. J. van Gerven. Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. Journal of Neuroscience , 35 (27):10005-10014, 2015. ISSN 0270-6474. doi: 10.1523/JNEUROSCI.5023-14.2015. URL https://www.jneurosci.org/content/35/27/10005 .

Tomoyasu Horikawa, Alan S. Cowen, Dacher Keltner, and Yukiyasu Kamitani. The Neural Representation of Visually Evoked Emotion Is High-Dimensional, Categorical, and Distributed across Transmodal Brain Regions. iScience , 23(5), May 2020. ISSN 2589-0042. doi: 10.1016/ j.isci.2020.101060. URL https://doi.org/10.1016/j.isci.2020.101060 . Publisher: Elsevier.

D. H. Hubel and T. N. Wiesel. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. The Journal of physiology , 160(1):106-154, January 1962. ISSN 0022-3751 1469-7793. doi: 10.1113/jphysiol.1962.sp006837. Place: England.

Alexander G. Huth, Wendy A. De Heer, Thomas L. Griffiths, Frédéric E. Theunissen, and Jack L. Gallant. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature , 532 (7600):453-458, April 2016. ISSN 0028-0836, 1476-4687. doi: 10.1038/nature17637. URL https://www.nature.com/articles/nature17637 .

Alexander G. Huth, Shinji Nishimoto, An T. Vu, and Jack L. Gallant. A Continuous Semantic Space Describes the Representation of Thousands of Object and Action Categories across the Human Brain. Neuron , 76(6):1210-1224, December 2012. ISSN 08966273. doi: 10. 1016/j.neuron.2012.10.014. URL https://linkinghub.elsevier.com/retrieve/pii/ S0896627312009348 .

Nidhi Jain, Aria Wang, Margaret M. Henderson, Ruogu Lin, Jacob S. Prince, Michael J. Tarr, and Leila Wehbe. Selectivity for food in human ventral visual cortex. Communications Biology 2023 6:1 , 6:1-14, 2 2023. ISSN 2399-3642. doi: 10.1038/s42003-023-04546-2.

Nancy Kanwisher, Josh McDermott, and Marvin M Chun. The fusiform face area: a module in human extrastriate cortex specialized for face perception. Journal of neuroscience , 17(11): 4302-4311, 1997.

Kendrick N Kay, Thomas Naselaris, Ryan J Prenger, and Jack L Gallant. Identifying natural images from human brain activity. Nature , 452(7185):352-355, 2008.

Alexander J. E. Kell, Daniel L. K. Yamins, Erica N. Shook, Sam V. Norman-Haignere, and Josh H. McDermott. A Task-Optimized Neural Network Replicates Human Auditory Behavior, Predicts Brain Responses, and Reveals a Cortical Processing Hierarchy. Neuron , 98(3):630644.e16, May 2018. ISSN 1097-4199 0896-6273. doi: 10.1016/j.neuron.2018.03.044. Place: United States.

Meenakshi Khosla, N. Apurva Ratan Murty, and Nancy Kanwisher. A highly selective response to food in human visual cortex revealed by hypothesis-free voxel decomposition. Current Biology , 32:1-13, 2022.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Ru Kong, Yan Rui Tan, Naren Wulan, Leon Qi Rong Ooi, Seyedeh-Rezvan Farahibozorg, Samuel Harrison, Janine D. Bijsterbosch, Boris C. Bernhardt, Simon Eickhoff, and B. T. Thomas Yeo. Comparison between gradients and parcellations for functional connectivity prediction of behavior. NeuroImage , 273:120044, June 2023. ISSN 1095-9572 1053-8119. doi: 10.1016/j.neuroimage.2023.120044. Place: United States.

Nikolaus Kriegeskorte. Deep neural networks: a new framework for modeling biological vision and brain information processing. Annual review of vision science , 1:417-446, 2015.

Nikolaus Kriegeskorte, Marieke Mur, Douglas A. Ruff, Roozbeh Kiani, Jerzy Bodurka, Hossein Esteky, Keiji Tanaka, and Peter A. Bandettini. Matching categorical object representations in inferior temporal cortex of man and monkey. Neuron , 60(6):1126-1141, Dec 2008. ISSN 08966273. doi: 10.1016/j.neuron.2008.10.043. URL https://doi.org/10.1016/j.neuron. 2008.10.043 .

Yuanning Li, Huzheng Yang, and Shi Gu. Upgrading voxel-wise encoding model via integrated integration over features and brain networks. bioRxiv , 2022. doi: 10.1101/2022.11.06.515387. URL https://www.biorxiv.org/content/early/2022/11/07/2022.11.06.515387 .

Yulong Liu, Yongqiang Ma, Wei Zhou, Guibo Zhu, and Nanning Zheng. Brainclip: Bridging brain and visual-linguistic representation via clip for generic natural visual stimulus decoding from fmri. arXiv preprint arXiv:2302.12971 , 2023.

Andrew Luo, Margaret Marie Henderson, Michael J. Tarr, and Leila Wehbe. Brainscuba: Fine-grained natural language captions of visual cortex selectivity. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum? id=mQYHXUUTkU .

Andrew F. Luo, Margaret M. Henderson, Leila Wehbe, and Michael J. Tarr. Brain Diffusion for Visual Exploration: Cortical Discovery using Large Scale Generative Models, November 2023. URL http://arxiv.org/abs/2306.03089 . arXiv:2306.03089 [cs].

Takuya Matsuyama, Shinji Nishimoto, and Yu Takagi. LaVCa: LLM-assisted Visual Cortex Captioning, February 2025. URL http://arxiv.org/abs/2502.13606 . arXiv:2502.13606 [q-bio].

Gregory McCarthy, Aina Puce, John C Gore, and Truett Allison. Face-specific processing in the human fusiform gyrus. Journal of cognitive neuroscience , 9(5):605-610, 1997.

Marieke Mur, Douglas A. Ruff, Jerzy Bodurka, Peter De Weerd, Peter A. Bandettini, and Nikolaus Kriegeskorte. Categorical, yet graded - single-image activation profiles of human category-selective cortical regions. Journal of Neuroscience , 32(25):8649-8662, 2012. ISSN 0270-6474. doi: 10.1523/JNEUROSCI.2334-11.2012. URL https://www.jneurosci.org/ content/32/25/8649 .

Till Nierhaus, Sara Wesolek, Daniel Pach, Claudia M. Witt, Felix Blankenburg, and Timo T. Schmidt. Content Representation of Tactile Mental Imagery in Primary Somatosensory Cortex. eNeuro , 10(6):ENEURO.0408-22.2023, June 2023. ISSN 2373-2822. doi: 10.1523/ENEURO. 0408-22.2023. Place: United States.

Toshiki Okumura, Ikuhiro Kida, Atsushi Yokoi, Tomoya Nakai, Shinji Nishimoto, Kazushige Touhara, and Masako Okamoto. Semantic context-dependent neural representations of odors in the human piriform cortex revealed by 7T MRI. Human brain mapping , 45(6):e26681, April 2024. ISSN 1097-0193 1065-9471. doi: 10.1002/hbm.26681. Place: United States.

OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mohammad Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman,

Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Jan Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O'Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov, Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr H. Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine B. Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. Gpt-4 technical report, 2024. URL https://arxiv.org/abs/2303.08774 .

Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Marc Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, Mahmoud Assran, Nicolas Ballas, Wojciech Galuba, Russell Howes, Po-Yao Huang, Shang-Wen Li, Ishan Misra, Michael Rabbat, Vasu Sharma, Gabriel Synnaeve, Hu Xu, Hervé Jegou, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. DINOv2: Learning Robust Visual Features without Supervision, February 2024. URL http://arxiv.org/abs/2304.07193 . arXiv:2304.07193 [cs].

Marius V. Peelen and Paul E. Downing. Selectivity for the human body in the fusiform gyrus. Journal of neurophysiology , 93(1):603-608, January 2005. ISSN 0022-3077. doi: 10.1152/jn.00513.2004. Place: United States.

Francisco Pereira, Bin Lou, Brianna Pritchett, Samuel Ritter, Samuel J. Gershman, Nancy Kanwisher, Matthew Botvinick, and Evelina Fedorenko. Toward a universal decoder of linguistic meaning from brain activation. Nature Communications , 9(1):963, March 2018.

ISSN 2041-1723. doi: 10.1038/s41467-018-03068-4. URL https://doi.org/10.1038/ s41467-018-03068-4 .

Paweł A Pierzchlewicz, Konstantin F Willeke, Arne F Nix, Pavithra Elumalai, Kelli Restivo, Tori Shinn, Cate Nealley, Gabrielle Rodriguez, Saumil Patel, Katrin Franke, et al. Energy guided diffusion for generating neurally exciting images. In Proceedings of the 37th International Conference on Neural Information Processing Systems , pages 32574-32601, 2023.

Xuelin Qian, Yun Wang, Jingyang Huo, Xinwei Sun, Yanwei Fu, and Jianfeng Feng. fMRI-PTE: A large-scale fMRI pretrained transformer encoder for multi-subject brain activity decoding, 2024. URL https://openreview.net/forum?id=BZkKMQ25Z7 .

N Apurva Ratan Murty, Pouya Bashivan, Alex Abate, James J DiCarlo, and Nancy Kanwisher. Computational models of category-selective brain regions enable high-throughput tests of selectivity. Nature communications , 12(1):5540, 2021.

Alexander Schaefer, Ru Kong, Evan M Gordon, Timothy O Laumann, Xi-Nian Zuo, Avram J Holmes, Simon B Eickhoff, and B T Thomas Yeo. Local-Global Parcellation of the Human Cerebral Cortex from Intrinsic Functional Connectivity MRI. Cerebral Cortex , 28(9):30953114, September 2018. ISSN 1047-3211, 1460-2199. doi: 10.1093/cercor/bhx179. URL https://academic.oup.com/cercor/article/28/9/3095/3978804 .

Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J Majaj, Rishi Rajalingham, Elias B Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Franziska Geiger, et al. Brain-score: Which artificial neural network for object recognition is most brain-like? BioRxiv , page 407007, 2018.

Rebecca F. Schwarzlose, Chris I. Baker, and Nancy Kanwisher. Separate face and body selectivity on the fusiform gyrus. The Journal of neuroscience : the official journal of the Society for Neuroscience , 25(47):11055-11059, November 2005. ISSN 1529-2401 0270-6474. doi: 10.1523/JNEUROSCI.2621-05.2005. Place: United States.

Paul S Scotti, Mihir Tripathy, Cesar Kadir Torrico Villanueva, Reese Kneeland, Tong Chen, Ashutosh Narang, Charan Santhirasegaran, Jonathan Xu, Thomas Naselaris, Kenneth A Norman, et al. Mindeye2: Shared-subject models enable fmri-to-image with 1 hour of data. arXiv preprint arXiv:2403.11207 , 2024.

Justine Sergent, Shinsuke Ohta, and Brenna Macdonald. Functional Neuroanatomy of Face and Object Processing: A Positron Emission Tomography Study. Brain , 115(1):15-36, February 1992. ISSN 0006-8950. doi: 10.1093/brain/115.1.15. URL https://doi.org/10.1093/ brain/115.1.15 . _eprint: https://academic.oup.com/brain/article-pdf/115/1/15/836448/1151-15.pdf.

Irina Simanova, Peter Hagoort, Robert Oostenveld, and Marcel A. J. van Gerven. ModalityIndependent Decoding of Semantic Information from the Human Brain. Cerebral Cortex , 24 (2):426-434, February 2014. ISSN 1047-3211. doi: 10.1093/cercor/bhs324. URL https: //doi.org/10.1093/cercor/bhs324 .

W. Kyle Simmons, Alex Martin, and Lawrence W. Barsalou. Pictures of appetizing foods activate gustatory cortices for taste and reward. Cerebral cortex (New York, N.Y. : 1991) , 15(10): 1602-1608, October 2005. ISSN 1047-3211. doi: 10.1093/cercor/bhi038. Place: United States.

Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. bioRxiv , pages 2022-11, 2022.

Jerry Tang, Meng Du, Vy Vo, Vasudev Lal, and Alexander Huth. Brain encoding models based on multimodal transformers can transfer across language and vision. Advances in neural information processing systems , 36:29654-29666, 12 2023.

Polina Turishcheva, Paul G Fahey, Michaela Vystrˇ cilová, Laura Hansel, Rachel Froebe, Kayla Ponder, Yongrong Qiu, Konstantin F Willeke, Mohammad Bashiri, Eric Wang, et al. The dynamic sensorium competition for predicting large-scale mouse visual cortex activity from videos. ArXiv , pages arXiv-2305, 2024.

L. N. van der Laan, D. T. D. de Ridder, M. A. Viergever, and P. A. M. Smeets. The first taste is always with the eyes: a meta-analysis on the neural correlates of processing visual food cues. NeuroImage , 55(1):296-303, March 2011. ISSN 1095-9572 1053-8119. doi: 10.1016/j.neuroimage.2010.11.055. Place: United States.

A. van der Schaaf and J.H. van Hateren. Modelling the power spectra of natural images: Statistics and information. Vision Research , 36(17):2759-2770, 1996. ISSN 0042-6989. doi: https://doi.org/10.1016/0042-6989(96)00002-8. URL https://www.sciencedirect.com/ science/article/pii/0042698996000028 .

Edgar Y. Walker, R. James Cotton, Wei Ji Ma, and Andreas S. Tolias. A neural basis of probabilistic computation in visual cortex. Nature neuroscience , 23(1):122-129, January 2020. ISSN 1546-1726 1097-6256. doi: 10.1038/s41593-019-0554-5. Place: United States.

Daniel L. K. Yamins and James J. DiCarlo. Using goal-driven deep learning models to understand sensory cortex. Nature Neuroscience , 19(3):356-365, Mar 2016.

Huzheng Yang, James Gee, and Jianbo Shi. Brain decodes deep nets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 23030-23040, 2024.

Jacob Yeung, Andrew F Luo, Gabriel Sarch, Margaret M Henderson, Deva Ramanan, and Michael J Tarr. Neural representations of dynamic visual stimuli. arXiv preprint arXiv:2406.02659 , 2024.

A Technical Appendices and Supplementary Material

A.1 Social impacts

In this work, we present a method for in silico mapping of semantic selectivity across cortical parcels in the whole brain. For cognitive neuroscience, our approach deepens the field's understanding of visual processing well beyond classical visual cortex. Clinically, deviations from normative whole-brain semantic maps may act as early biomarkers for neurological and neurodegenerative disorders. Furthermore, these detailed semantic atlases lay foundational groundwork for next-generation brain-computer interfaces, where fine-grained category-level decoding could translate a user's intended concepts-rather than low-level motor signals-into control commands for communication or assistive devices.

A.2 Reproducing selectivity for known areas

We show superstimuli from parcels overlapping with body-, place-, and word-selective areas in order to show that our pipeline can reproduce the known selectivity.

Natural Scenes Dataset

Generated

Imagenet

Natural ScenesDataset

Generated

Imagenet

NaturalScenesDataset

Generated

Imagenet

190.51

MEN

ACEYAN

AWEEHP.Y

LAETEA

(HO)

TUOVES

:EAC

四

DIRTY!

SKATER

AIR

THeN

SPRAY

STOP

Dtisod

UUDONAE

CECKIZSHTS

XCTCEY.

JHEEN

Rolmoive

tla-U/PUTEM

ws lnoal

CAS

IBN

VIYES

TOREY

GAT'N

PNR

COMPAG

CORVAIR

OILFILTER

C47C

1810X

VIGOR

7-UD

AVLFE

GOD

CNJ

SLSO

neue

STOP

VLAPR

B

TRATAMIENTE

LALAR

RobbyKriege

www.

.com

whitening,

HIVEA

body

UV

BOANA

A.3 Parcel selection process

ject1lefthemisphereparcelselection

(b)

(c)

Qualifying parcels by prediction accuracy

Selected parcelsby noise ceiling and prediction accuracy

off

24

cutoff

0.7

Nonselected

21

Parcel mean model prediction accuracy

0.6

Noise ceiling

Selected

18

0.5

Count

15

0.4

12

0.3

6

0.2

6

0.1

3

0.0

0

0.1

0.2

0.3

0.4

0.5

0.6

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Parcel mean model prediction accuracy

Parcel mean noise ceiling (normalized)

A.4 Reproducing known visual hierarchy properties

We optimized 32 superstimuli (16 per hemisphere) that maximally activate parcels in V1, V2, and V4 using the BrainDIVE framework. We additionally include results on FFA as a representative IT-cortex ROI.

Generated superstimuli for early visual areas reproduce the classic fine-to-coarse hierarchy: V1 stimuli appear as cluttered scenes filled with dense, repetitive texture; V2 images add composite color patches and rudimentary objects; V4 stimuli reveal smoother, recognizable object forms; and FFA images almost exclusively depict close-up faces, often with clear emotional expressions.

To quantify hierarchical properties, we examined whether the spatial-frequency content of our stimuli mirrors classical physiological findings (e.g., [31, 18]). We computed the radial average power spectrum for each image [70] and calculated the proportion of spectral power above 10%, 20%, and 30% of the maximum spatial frequency. At every threshold the high-frequency energy ratio decreases monotonically from V1 > V2 > V4 > FFA, indicating that the images that best drive higher-level areas (FFA) contain proportionally less fine-scale texture and relatively more coarse, low-frequency structure.

Below we include superstimuli optimized for each of the above ROIs to visualize the image frequency differences.

(a) V1

(b) V2

(c) V4

(d) FFA-1

A.5 Analyzing ROI queries to map parcel connectivity

Taking advantage of our transformer-based approach, we map the connections between visual area parcels by examining the representational similarity of learned ROI queries, each of which corresponds to a single parcel. For both functional connectivity and ROI query similarity, we first take the full 1000 parcel x 1000 parcel matrix, then mask out the labeled area, and finally average across the top three parcels that overlap with each labeled area. (Thus, not every entry on the diagonal is 1.) High cosine similarity between ROI queries (queries averaged across ensemble models) suggests that two parcels are highly connected, as both attend to similar content in an image.

This similarity matrix for the visual areas closely replicates the functional connectivity matrix from the Schaefer parcellation (derived from resting-state correlated responses) [40]; the Pearson correlation between the upper triangle of the two matrices is ≈ 0 . 5 .

Below we report the functional connectivity and ROI query cosine similarity matrices for subject 1.

LH-V1v

LH-V1d

LH-V2v

LH-V2d

LH-V3v

LH-V3d

LH-hV4

LH-EBA

LH-FBA-1

LH-OFA

LH-FFA-1

LH-OPA

LH-PPA

LH-RSC

LH-OWFA

LH-VWFA-1

LH-VWFA-2

LH-mfs-words

Parcel

LH-V1v

LH-V1d

LH-V2v

LH-V2d

LH-V3v

LH-V3d

LH-hV4

LH-EBA

LH-FBA-1

LH-OFA

LH-FFA-1

LH-OPA

LH-PPA

LH-RSC

LH-OWFA

LH-VWFA-1

LH-VWFA-2

LH-mfs-words

Parcel

Parcel Functional Connectivity Matrix

0.0

0.1

0.2

0.3

0.4

0.5

0.6

Functional Connectivity

(a) Functional connectivity, left hemisphere

LH-V1v

LH-V1d

LH-V2v

LH-V2d

LH-V3v

LH-V3d

LH-hV4

LH-EBA

LH-FBA-1

LH-OFA

LH-FFA-1

LH-OPA

LH-PPA

LH-RSC

LH-OWFA

LH-VWFA-1

LH-VWFA-2

LH-mfs-words

Parcel

LH-V1v

LH-V1d

LH-V2v

LH-V2d

LH-V3v

LH-V3d

LH-hV4

LH-EBA

LH-FBA-1

LH-OFA

LH-FFA-1

LH-OPA

LH-PPA

LH-RSC

LH-OWFA

LH-VWFA-1

LH-VWFA-2

LH-mfs-words

Parcel

ROI Query Similarity Matrix

0.00

0.05

0.10

0.15

0.20

0.25

0.30

Cosine Similarity

(b) ROI query similarity, left hemisphere

RH-V1v

RH-V1d

RH-V2v

RH-V2d

RH-V3v

RH-V3d

RH-hV4

RH-EBA

RH-FBA-1

RH-OFA

RH-FFA-1

RH-FFA-2

RH-OPA

RH-PPA

RH-VWFA-2

Parcel

RH-V1v

RH-V1d

RH-V2v

RH-V2d

RH-V3v

RH-V3d

RH-hV4

RH-EBA

RH-FBA-1

RH-OFA

RH-FFA-1

RH-FFA-2

RH-OPA

RH-PPA

RH-VWFA-2

Parcel

Parcel Functional Connectivity Matrix

0.0

0.1

0.2

0.3

0.4

Functional Connectivity

(c) Functional connectivity, right hemisphere

RH-V1v

RH-V1d

RH-V2v

RH-V2d

RH-V3v

RH-V3d

RH-hV4

RH-EBA

RH-FBA-1

RH-OFA

RH-FFA-1

RH-FFA-2

RH-OPA

RH-PPA

RH-VWFA-2

Parcel

RH-V1v

RH-V1d

RH-V2v

RH-V2d

RH-V3v

RH-V3d

RH-hV4

RH-EBA

RH-FBA-1

RH-OFA

RH-FFA-1

RH-FFA-2

RH-OPA

RH-PPA

RH-VWFA-2

Parcel

ROI Query Similarity Matrix

0.00

0.05

0.10

0.15

0.20

0.25

0.30

0.35

Cosine Similarity

(d) ROI query similarity, right hemisphere

A.6 Attention maps

While one can easily make guesses about parcel selectivity simply by looking at the prominent feature of the image, we can examine the attention scores from cross-attention (Figure 1b) to interpret the selectivity of any parcel. The attention maps are visualized in Figure S4 for the images from Figure 5.

Natural Scenes Dataset

Generated

Imaaenet

Natural Scenes Dataset

Generated

Imaaenet

AWXEMP.Y

LARTXA

(HO)

DIRTY!

SETE

AIR

THNNER

STOP

TUOVES

:EAC

Dfiosod

As expected, the areas with highest attention weights for the aTL-faces parcel primarily overlap with faces, while the attention weights for the mfs-words parcel overlap with words (when present). Compared to previous work using a similar encoder model [3], our attention maps appear to follow the expected stimuli considerably less closely, possibly because our parcel boundaries were determined a priori and don't align well with the boundaries of the actual labeled area.

A.7 Parcel locations and BrainDIVE/NSD images accompanying Figure 1

PS)

es1)

IPSO

V3A

K02

V4

OPA

01

PPA

(Ps1

IPSO

V3A

LoZ

K02

A.8 Labeling top subject-specific parcels

For each of the parcels shown in Figure 6, we show the top ImageNet images and the caption generated by GPT-4o (see Appendix A.18 for prompt details).

Parcel

ImageNet

top 4

Caption

Sports, Tennis,

Soccer

Soccer, Tennis,

Sports

Sports, Tennis,

Soccer

Babies, Children,

Family

Family, Parenting,

Children

Food storage,

Office supplies,

Kitchen appliances

S2 RH 391

Gymnastics,

Dance, Tennis

Tennis,

Gymnastics,

Baseball

Social Gatherings,

Education and

Learning, Sports

and Recreation

Interior Design,

Living Spaces,

Furniture

Parcel

ImageNet

top 4

Caption

S5 RH 476

Sports, Team

sports, Outdoor

activities

S5 LH 239

Extreme sports,

Skateboarding,

BMXbiking

S5 LH 367

Interior Design,

Kitchen

S5 RH 391

Soccer, Ice hockey,

Tennis

S5 LH 167

Kitchen,

Appliances,

Cabinets

Tennis, Baseball,

Sports

Tennis, Baseball,

Sports

Sports, Baseball,

Tennis

Baseball, Pitching,

Batting

S7 RH 317

Baseball, Pitching,

Batting

KIA

A.9 Full collages for cross-subject tools parcel

(c) Subject 5

(d) Subject 7

A.10 Statistical tests for subject-specific parcel evaluation

Here, we explain the details of the statistical tests performed in 4.4.

The underlying assumption in cognitive neuroscience is that for some parcels, there exists some category of visual stimuli that maximally activates the parcel.

Our goal is to assess whether the semantic content of the images our encoder identifies as highly activating aligns with the semantic content of NSD test images that actually elicit high ground-truth activations in that parcel. We ask whether a single 'concept vector' derived from our encoder for a given parcel predicts parcel-activation rank order in the NSD test set more accurately than the baseline model.

We computed two concept vectors per parcel: one based on the top 32 maximally-activating ImageNet or BrainDIVE images predicted by our encoder, and one based on the top 32 maximally-activating NSD training images. We embedded each set of the images into CLIP space and then averaged the CLIP vectors to obtain a parcel-specific "concept vector" that reflects predicted parcel categorical

selectivity. We then embedded each NSD test image into CLIP space, and ranked images by cosine similarity to the concept vector (encoder-derived or baseline). Since we expect that higher semantic similarity to the 'concept vector' will correspond to higher parcel activation, we quantified alignment with the empirical activation ordering using Spearman's rank correlation between concept-vector similarity rank and ground-truth fMRI activity rank.

In Table 2, for each parcel, we compared the rank correlation coefficient with a null distribution, bootstrapped with 10,000 random rankings of the NSD test images. We separate the results into most activating stimuli predicted by the model from BrainDIVE and ImageNet, as well as stimuli from the NSD training set (baseline).

A.11 Statistical tests for cross-subject parcel evaluation

For a given parcel and dataset, we generated a selectivity hypothesis by averaging the CLIP vectors corresponding to the top 32 images according to the brain encoder, then used that CLIP vector to rank the NSD training images using cosine similarity. We calculate Spearman rank correlation between that ranking and the ranking determined by ground truth parcel activation, and compare this to a null distribution of correlation coefficients, which consists of 10,000 draws of random rankings.

Note that there are a fewer number of total parcels measured here since each subject has different parcels selected for experimentation (see Section 3.3), so cross-subject selectivity can only be tested for parcels selected for experimentation with every subject.

A.12 Cosine similarity analysis of labels for cross-subject parcels

To demonstrate cross-subject consistency in semantic selectivity, we also compared directly the concept vectors generated by the encoder. Concept vectors generated for the same parcel, when analyzed across subjects, tend to be more similar (0.784 ± 0.069 for BrainDIVE) than the parcels coming from around the target parcel (0.710 ± 0.115 for BrainDIVE). Absolute cosine similarities are high because these parcels are selective for high-level visual concepts that share similar features, but our method can adjudicate between them using targeted stimulus generation.

A.13 Head-to-head encoder model versus NSD train baseline

Table 8 expands upon the results shown in table 2, reporting the fraction of parcels for which the encoder-selected stimuli outperforms the NSD train baseline in Spearman's rank correlation coefficient, evaluated within the same subject.

Table 9 expands upon results in Table 4, reporting the number of parcels for which the rank correlation coefficient from the specified model is greater than that from the baseline model, when three subjects are used to predict a held-out subject.

A.14 Choosing parcels shared across subjects for future fMRI experimentation

In Figure S8, we report similar results as in Figure 6, except for top parcels shared across subjects.

2 5

2 6

2 7

2 8

2 9

2 10

2 11

2 12

2 13

Number of Retrieved Images

0.0

0.2

0.4

0.6

0.8

1.0

Subject 1 predicted by 2/5/7

LH 203

RH 195

LH 315

RH 179

LH 170

95% CI

2 5

2 6

2 7

2 8

2 9

2 10

2 11

2 12

2 13

Number of Retrieved Images

0.0

0.2

0.4

0.6

0.8

1.0

Subject 2 predicted by 1/5/7

LH 185

LH 170

LH 187

RH 175

LH 217

95% CI

2 5

2 6

2 7

2 8

2 9

2 10

2 11

2 12

2 13

Number of Retrieved Images

0.0

0.2

0.4

0.6

0.8

1.0

Subject 5 predicted by 1/2/7

LH 363

LH 217

RH 237

RH 338

LH 496

95% CI

2 5

2 6

2 7

2 8

2 9

2 10

2 11

2 12

2 13

Number of Retrieved Images

0.0

0.2

0.4

0.6

0.8

1.0

Subject 7 predicted by 1/2/5

RH 477

RH 343

LH 363

RH 195

RH 237

95% CI

NSD Train Imagenet Retrieval Rate

A.15 Interpreting statistical test results by comparing to known visual areas

To quantify what rank correlation magnitude constitutes meaningful selectivity, we report the rank correlation the model achieves on known visual areas as a benchmark. Parcels with known highlevel selectivity (EBA, FFA, FBA) tend to exhibit very high correlations, while parcels with known lower-level selectivity (V1-V4) tend to exhibit low correlations. The average Spearman's ρ from our concept vectors are in the range of areas like PPA and RSC.

Using CLIP as our metric allows us to capture high-level features that can explain semantic selectivity. Given that fMRI studies tend to rely on experimenter-curated topics, the CLIP space is a move toward a data-driven approach, but ultimately any metric can be substituted.

Figure S9: Spearman's ρ , actual vs. predicted activation ordering for subject 1. Top: high-level areas. Bottom: low-level areas.

(a) High-level areas

(b) Low-level areas

Note that since we're using the Schaefer-1000 parcellation, for each visual area, we average the rank correlation value across the top 3 Schaefer parcels with greatest overlap with that visual area.

A.16 Varying top images used for statistical test

To confirm that the superior performance of ImageNet and BrainDIVE superstimuli stems from a broader diversity of concepts present-and not sampling near peak activation-we varied the top-K image threshold used to define the concept vector. We report K = 16 , 8 , 4 , 2 , 1 versions for tables 2, 3, 4, and 5.

A.17 Generating concept vectors using predicted activations

In the main text, we generated the baseline NSD train concept vector using the top maximallyactivating images based on measured responses of the training set. Here, we generate concept vectors using top images based on predicted responses of the same training set-reproducing tables 2, 3, 4, and 5. Interestingly, the rankings generated by the model-predicted concept vector outperforms the concept vector generated by measured responses, and even our model in some cases. We suspect that using model-predicted responses may further reduce the noise compared to measured responses and be responsible for the increase in performance. We did not include the model-predicted concept vector performance in the main text since it's not quite a baseline (since it relies on the encoding model choice).

Notes. INet = ImageNet. BD = BrainDIVE.

Notes. INet = ImageNet. BD = BrainDIVE.

A.18 LLMprompts used

We used OpenAI's GPT-4o [52] to caption the images selected by our model in order to obtain a linguistic representation of the selectivity of a parcel. The captions in Figure 7 were generated with ChatGPT 4o (04/06/2025) with a collage of top-25 encoder-ranked ImageNet images shown, followed by the instruction 'Describe a theme present in most of the images presented.' The captions in Table 7 were generated by GPT-4o (05/12/2025, via the API) with the prompt 'Give keywords for the central concept or categories present in these images.'

A.19 NSD dataset quirks

A.20 Compute used

We used CPU (AMD EPYC 7662), GPU (NVIDIA A40, L40), memory, and storage resources from an internal cluster. Storage for the entire project totals roughly 10TB. Training the model used roughly 3,000 GPU hours, 24,000 CPU core hours, and 32 GB per GPU hour. Running the remaining experiments used roughly 20,000 GPU hours, 160,000 CPU core hours, and 32 GB per GPU hour. The full project required more compute than these estimates due to failed experiments, experiments not included in the paper, and model iteration.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: In the abstract, we explain our approach and the results, include parcels of which we labeled the selectivity and our statistical tests that verify our labels. The Section 3 Methods explains the implementation details for our model. Section 4 Experiments show examples of parcels that we labeled and the results from the statistical tests that show our labels do drive the activation of the parcels.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: Limitations are discussed in Section 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: Information about the datasets, model architecture, and steps to perform statistical tests are all contained in the paper.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The supplementary zip file contains open access to the data and code with sufficient documentation.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: Training and test details can be found in the supplementary zip file containing the code to reproduce the experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: Error bars and p-values are reported in all tables. Details for implementation of the statistical tests are found in the main text and appendix.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: See Appendix A.20 for estimates on computer resources used.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conforms in every respect with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The paper's societal impacts are discussed in Appendix A.1.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The paper cites any data and models used and respects the respective licenses.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: Anonymized assets will be provided in the supplementary zip. Materials will include documentation.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [No]

Justification: The core method in this research does not involve LLMs as any important, original, or non-standard components.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a transformer-based encoder-decoder model ensemble to predict fMRI parcel vertex activity from image stimuli and use this as an in-silico model of the brain to study visual selectivity of cortical regions.  The introduced ensemble is composed of multiple frozen image encoder backbones and trainable decoders to predict fMRI responses.  The authors demonstrate the use of this ensemble model in assigning semantic meaning to parcels in and outside of the visual cortex (within single subjects and across multiple subjects) as a means to efficiently generate hypotheses about categorically selective regions of the brain.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- Practical Utility in Experimental Design:  The paper presents an intuitive framework for selecting brain parcels for future fMRI studies, particularly through its “Choosing parcels for future fMRI experimentation” section. This provides experimenters with a principled and cost-effective method to prioritize regions for follow-up studies based on encoder-derived selectivity metrics.\n- Mapping selectivity beyond classical visual areas:  The paper offers proof-of-concept results (proof-of-concept in the sense that in some cases selectivity hypotheses were generated by the model but not verified in-vivo) for identifying selectivity of regions outside of canonical visual areas. This type of analysis could offer insight into integration of visual information into regions of the brain that are not directly involved in visual processing.\n- Quantitative evaluation of hypotheses:  The authors go beyond qualitative image analysis and validate parcel selectivity hypotheses using semantic similarity metrics and ground-truth fMRI data.\n\nWeaknesses:\n- Positioning relative to prior work: While the paper introduces an ensemble transformer model, many prior works have also trained DNNs on the NSD dataset to predict brain responses and use these models to infer selectivity of arbitrary voxels.  It is unclear to me how this method substantially differs in outcome or capability beyond architectural novelty.\n- Lack of baseline comparisons: A more direct comparison between existing models that predict fMRI activity from image stimuli or ablation would help clarify the need for the proposed model ensemble (Does the proposed model predict fMRI activity more accurately than existing methods? Are the learnable parcel queries necessary compared to vanilla decoders?).  The authors suggest that previous works are limited by the fact that they frequently rely on the use of an affine transformation from model representations to predict fMRI activity, but do not seem to provide evidence that their proposed model offers more accurate predictions.\n- Method is limited to regions of the brain that are categorically and semantically selective:  While the model demonstrates semantic selectivity for higher-order concepts and object categories, it does not convincingly show whether it can uncover selectivity for lower-level visual features (e.g., texture, orientation, motion), which are critical components of the visual processing hierarchy.\n\nQUESTIONS:\n- Ensemble ablation: What is the variability in fMRI predictivity among the individual models of the ensemble?\n- Comparison to existing baselines: How much better does the proposed ensemble (and also individual models in the ensemble) predict fMRI activity compared to pre-trained vision (e.g., ResNet-50, DINO, etc.) models with learned affine transformations from intermediate layers?\n- Comparison with existing work: Where do you suggest this method stands in comparison to pre-trained models that have been aligned with cortical regions of the brain with affine transformations and diffusion-based methods (e.g., MindEye) that synthesize images based on in-silico fMRI activity (thereby offering a platform for analyzing selectivity of different brain region)? Any concrete examples of what the proposed method can reveal and what existing methods cannot would help clarify the contributions of this work.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-27"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis manuscript demonstrates an in-silico approach for mapping visual categorical selectivity across the whole brain, moving beyond traditional methods that rely on controlled stimuli and linear encoding models. By leveraging an encoder-decoder transformer with a brain-region to image-feature cross-attention mechanism, the proposed architecture flexibly aligns high-dimensional deep network features with cortical responses. Coupled with diffusion-based image generative models and large image datasets, this method can synthesize and select images that maximize the activation of different cortical parcels, revealing regions with complex compositional selectivity involving diverse semantic concepts. The findings suggest a new approach for discovering visual selectivity, offering a more flexible and scalable alternative to traditional fMRI experiments for hypothesizing and testing visual concepts.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\nThis manuscript introduces an in-silico approach using an encoder-decoder transformer with a cross-attention mechanism, capable of aligning high-dimensional deep network features with cortical responses, thereby enabling the synthesis and selection of images that maximally activate different cortical parcels without controlled stimuli and linear encoding models.\n\n**Weaknesses**\n\nAlthough this research offers an interesting approach to identifying superstimuli, the lack of empirical validation of these superstimuli in new fMRI experiments inherently highlights a weakness.\n\nQUESTIONS:\n**Major concerns**\n\n1. Although this research provides an interesting approach to finding superstimuli, the lack of empirical validation of these 'superstimuli' through new fMRI experiments is a clear limitation. However, I do not consider this an essential prerequisite for the paper's acceptance. To further bolster the model's persuasiveness without immediate empirical validation, more robust evidence is needed. I believe the paper's convincingness would be significantly enhanced by providing richer examples akin to Figure 4 (Verifying the selectivity of aTL-faces). Could similar examples be generated for other previously identified fMRI-sensitive areas such as the PPA (Place-selective), EBA (Body-selective), and VWFA (Word-selective)?\n\n2. Beyond merely predicting categorical selective areas, is it possible to analyze changes in characteristics along the visual hierarchy? For instance, if 'superstimuli' could be identified along the ventral visual stream from V1 to IT, and these demonstrated previously known hierarchical processing properties, it would offer another robust validation point for the model.\n\n3. The paper claims to provide interpretability through its cross-attention mechanism, specifically arguing that \"the areas with highest attention weights for the aTL-faces parcel primarily overlap with faces, while the attention weights for the mfs-words parcel overlap with words (line 537).\" However, Figure 8 appears to show attention weights that are merely high at the center of the image, rather than precisely aligning with the positions of faces and words. How can you more rigorously verify that the attention mechanism truly focuses on the selective object (face or word) rather than simply a central region?\n\n4. To more directly assess the object-specific attention effect of your cross-attention mechanism, consider presenting multi-cue images that contain both a face and a word within the same image. How would you then visualize and analyze the attention weights specifically for the aTL-faces and mfs-words parcels in this scenario? This could provide clearer evidence of selective attention.\n\n**Minor issues**\n\n1. Figure 4 captions: last (b) actually means (e)?\n\n2. line 539: “previous work using a similar encoder model [] “ -> Empty reference\n\n3. line 890: “The paper doe snot\" should be corrected to “does not”\n\nLIMITATIONS:\nThis manuscript proposes a model for mapping visual categorical selectivity; however, the lack of experimental validation for the suggested 'superstimuli' inherently limits the plausibility of the model.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a pipeline for characterizing semantic selectivity across the entire cortex using fMRI responses to natural images. The method relies on a brain encoder that predicts voxel-wise responses from images, from which a “concept vector” is constructed per cortical parcel by averaging CLIP embeddings of top-activating images. Semantic selectivity is then evaluated via Spearman correlation between parcel activation predictions and CLIP-based similarity scores. The approach is used to compare different image sources (NSD-train, ImageNet, BrainDIVE), and is extended to assess cross-subject generalization of semantic selectivity.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n- Technically sound integration of encoding models and CLIP-based embeddings into a pipeline to assess semantic selectivity.\n- Explores semantic selectivity beyond canonical visual areas, using the large-scale NSD dataset.\n- Encoder-agnostic framework with potential applicability to other predictive models (although not demonstrated).\n- Evaluation includes cross-subject generalization, which strengthens the methodological contribution.\n\n**Weaknesses**\n\n- At times I found it hard to follow the methodological setup, particularly regarding how the concept vectors were constructed in different conditions (e.g., for cross-subject experiments), whether actual or predicted responses were used for NSD-train, and what exactly the authors consider part of the \"pipeline\". These ambiguities made it difficult to interpret the core comparisons and contributions.\n- Reported Spearman correlations are low, and no threshold is provided to interpret what constitutes meaningful semantic selectivity.\n- It is not clear whether the differences observed across image sets is actually due to concept coverage or could it be due to sampling density around activation peaks.\n- Several implementation details (e.g., whether measured or predicted responses were used, how concept vectors are constructed for the cross-subject analysis) were not clear to me.\n- Conceptually, the work builds on existing ideas and may offer limited novelty beyond assembling and evaluating known components.\n\nQUESTIONS:\nGenerally, I think the work has potential and raises interesting ideas, but in its current form I have concerns about its conceptual clarity and novelty. If the authors can address the following questions, I’d be happy to reconsider and possibly increase my score.\n\n**1. Could the result in Table 2 simply reflect differences in input sampling density around activation peaks, rather than concept coverage?** in Table 2, the only difference between conditions is the \"concept vector\" used per parcel, computed as the average CLIP embedding of the top-K (K=32) activating images. Since a good concept vector depends on sampling near peak activation, image sets like BrainDIVE (explicitly optimized for this) and ImageNet (large and dense) naturally yield better vectors. In contrast, the NSD-train set is relatively small and not optimized for activation maximization, making it less likely to produce **representative concept vectors** — even if the underlying concepts are present. Thus, the performance gap may reflect sampling effectiveness rather than a difference in the actual set of concepts represented. The current analysis seems to conflate these, interpreting lower values for NSD-train as indicating fewer covered concepts. To support this claim more rigorously, it would help to control for sampling density — for example, by varying the top-K threshold used to define the concept vector. If NSD-train still underperforms at lower K (where the top images should be closest to the activation peak), that would strengthen the argument.\n\n**2. Same concern applies to Table 3**: Since the encoder-driven image sets yield higher spearman rank correlations the authors conclude that “brain-encoder pipeline can generate finer-grained categorical hypothesis” than images in the train set. Again, this is very likely a direct consequence of having a better estimate of a parcel’s concept vector, which itself depends on having enough images sampled around maximum activation for that parcel. Showing these results across different values of K (i.e., the number of top-activating images used to define the concept vector) would clarify whether the NSD-train set truly yields a worse selectivity hypothesis, or if the observed differences are simply due to fewer or less optimized samples contributing to the concept vector. This would help disentangle model differences from sampling effects.\n\n**3. What are some examples of follow up experiments for which the parcels with higher retrieval in Figure 5 are promising targets?** The metric rewards parcels where the concept vector retrieves the ground-truth top-activating images well. While this likely reflects some degree of semantic selectivity, it's not clear why this makes these parcels \"promising targets\" for follow-up fMRI studies. What would an experimenter test for with these parcels and/or the corresponding images that they seem to be semantically selective for (from ImageNet or BrainDIVE)?\n\n**4. Did the NSD-train condition use actual measured responses to select the top-K images, or were the selections based on model-predicted responses to training images (like ImageNet and BrainDIVE)?** This detail is important for interpreting the NSD-train condition as a proper baseline. If the encoder is also used for NSD-train, then all conditions reflect model-based estimates. If not, then the performance gap could partly reflect differences in response quality (e.g., measured vs. predicted) rather than conceptual coverage alone. Clarifying this would help in comparing conditions more fairly.\n\n**5. What is the main conceptual advancement beyond implementing an evaluation framework for comparing image sets?** The pipeline offers an approach to quantify semantic selectivity. The main message, as far as I understood, is that since the approach relies on highly exciting inputs to construct a parcel-specific “concept vector” using a good set of inputs matter. And the findings largely align with what one would expect: images optimized to be maximally activating (BrainDIVE) or images picked from a densely sampled set (ImageNet) naturally yield better performance than a limited training set. In addition, the paper also assesses cross-stimuli and cross-subject generalization, but I find those results hard to interpret (explained in the next question). Could the authors clarify which aspects, beyond the application of existing tools, constitute a novel conceptual or methodological contribution. Is it the use of CLIP to create “concept vectors” for parcels where their semantic selectivity is not known?\n\n**6. Does the correlation magnitude matter?** In both Tables 3 and 5, the reported Spearman correlations are rather low. Assuming the concept vectors are accurate and CLIP space reflects semantic similarity, strong semantic selectivity should yield high rank alignment (i.e., high $\\rho$). However, the paper provides no principled threshold or theoretical grounding for what magnitude of $\\rho$ reflects meaningful selectivity. How high should the correlation be for a parcel to be considered semantically selective, and how low for it to be weakly/not selective? As a reference point, it would be informative to show the same analysis on the known visual parcels - do they exhibit (noticeably) higher correlations?\n\n**7. Why not directly compare concept vectors of the same parcel across subjects to demonstrate cross-subject consistency in semantic selectivity?** While Tables 4 and 5 demonstrate that concept vectors derived from other subjects exhibit semantic selectivity in a target subject, this provides only indirect evidence for cross-subject generalization. It does not directly assess whether the semantic selectivity of corresponding parcels is similar across individuals. In Figure 6, the authors illustrate this point qualitatively using example images, but a more direct test would be to compute the pairwise cosine similarity between concept vectors of corresponding parcels across subjects. This would more explicitly support the assumption that the same parcel across subjects encodes similar semantic content. Have the authors performed such an analysis?\n\nOther (minor) points:\n\n- In section 4.5 how are the authors exactly constructing the “concept vector” using the encoder models trained on other subjects? Are you pooling top-32 maximally activating images across three models (i.e. those trained on other subjects)? Please provide some details on how the concept vector is constructed.\n- Is the encoder part of the pipeline and is the encoder a contribution of the paper? My initial understanding was not the pipeline is additional steps around the encoder, which aligns well with the statement in line 64: \"...our pipeline is ultimately encoder-agnostic, and can use any encoder that is image-computable\". But then the usage of the term “brain encoder pipeline” in line 224 seem to imply that the encoder is part of the pipeline. I would appreciate some clarification here, and I would also suggest to improve the text to potentially avoid such confusions.\n- Please refer to specific subplots/panels in Figure 3 both in the text (lines 141-144) and the caption. Also, please label the colorbars.\n- There is a repeated sentence in lines 222-223.\n\nLIMITATIONS:\nThe authors already point out some relevant limitations such as the need for experimental validation of “superstimuli” and the fact that encoding model can be biased due to the training set. I think these are fair and useful to acknowledge. A few more points might be worth considering:\n\n- The approach assumes that CLIP embedding space reflects something close to the brain’s semantic organization. That might hold in some regions (e.g., visual cortex), but it’s unclear how valid that assumption is more broadly.\n- The method is heavily based on top-K activation which could be problematic. For example, differences in performance across datasets could just reflect sampling density near activation peaks, not actual differences in concept coverage.\n- The Spearman correlation values used throughout (especially in Tables 3 and 5) are fairly low, and it’s unclear what values should be considered meaningful? Some sort of benchmark (e.g. visual cortex parcels) would help give these numbers context.\n- Tables 4 and 5 suggest cross-subject generalization, but they don’t directly test if the *same* parcels across subjects are encoding similar concepts. Comparing concept vectors for the same parcel across individuals (e.g. using cosine similarity) would provide a more clear evidence here.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors propose a data-driven in silico approach based on fMRI activation predictive encoding models that would allow for images predicted to yield high activations in the target area to be selected from a pool of images or be generated by using a diffusion-based decoder employing the encoder model’s gradient. Using this approach, the authors higlight that the method can indeed select out mostly face-containing images for aTL (area known for face selectivity) and further show that images selected by their approach tends to yield semantically meaningful cluster (as characterized by CLIP embedding) such that images closer to this embedding in the CLIP space tends to elicit strong response in the encoding model.\n\nSTRENGTHS AND WEAKNESSES:\nQuality\n* It appears to me that the encoding model-based stimulus selection as put forward by the authors in this work is conceptually very similar, if not identical, to what has been proposed and worked on over the last couple of years across electrophysiological and optophysiological data in NHP and mice. However, citations of this relevant literature appear to be grossly lacking.\n* The work offers very little novel technical contributions. While it would have been exciting to see whether the new image category selected for each parcel to be maximally activating would have actually led to higher activation, the work does not include experimental verification (the authors note this limitation). As the technique is not particularly novel (aside from possibly the technique being applied to fMRI dataset, rather than electrophysiolgy or optophysiology recordings), it would have been seminal to see the predictions tested in real experiments.\n* In the evaluation of the labels, the authors compare the degree to which the model-derived label predicts (ranks) parcel activation rankings better than chance as a measure of labeling consistency. Although the reasoning is intuitive, it is not clear whether the use of semantic proximity is always the most appropriate measure to assess the similarity of a test image to the labeled most activating image. One would imagine an appropriate measure of image similarity precisely depends on the exact visual features that the parcel is selective to, and as such, how well the semantic proximity (in their case, the proximity in the CLIP embedding) can assess the parcel-relevant feature similarity entirely depends on a priori unknown alignment of CLIP embedding and correspoinding image embedding based on the parcel activities. Furthermore, although encoder-based selections do appear to achieve higher fraction scores, the difference from the baseline is rather small. Without experimental verification, it is unclear if the baseline selection vs encoder-based selection would have yielded experimentally observable and meaningful difference in the activations.\n\n\nClarity\n* The term NSD was introduced before the first mention of Natural Scene Dataset appearing in L121 and even then explicit correspondence of the acronym was never made. Please be sure to introduce the full term at the first use of the acronym appearing on L112\n\n\nSignificance & Originarity \n* As the authors state, the use of large and naturalistic stimulus sets (such as naturalistic images) is critical in adequately studying the functions of stimulus-responsive neural populations. However,  the large number of trials required to present such datasets proves to be empirically prohibitive. In line with the recent trend, the authors identify AI models trained on an extensive collection of neural datasets as “digital twins,” which enables extensive in silico experiments. In this work, the authors introduce a pipeline framework through which a trained encoding model can be used for mapping visual categorical selectivity, which in turn can be used to generate hypotheses at the category/concept level.\n* While the proposed apporach of using a high-accuracy encoder model of the brain (in this work’s case, predicting vertices activation from image input) in selecting (from a held out natural images) or generating (based on generative model where the generation can be guided via gradiet of the predictive model) images that would most strongly activate the target brain areas is a powerful technique, this framework is by no means original as there has been numerous work exactly on this approach in the last ~6 years including the work of Bashivan, Kar & DiCarlo Science 2019, Walker, et al. NatNeuro 2019, and Perzchlewicz, et al. NeurIPS 2023, just to name a few.\n* Consequently, I’m not clear what particular novel contribution is made by the work. I fail to see the idea of using encoding model to predict highly activating stimulus to be conceptually novel (as this has been proposed with rigorous validation in the work cited above), and use of generative model guided by encoding model to generate high activating stimuli has also been addressed in the work of Perzchlwicz et al (NeurIPS 2023) with even experimental verification. Although it is possible that these techniques have not been used extensively in fMRI settings, I would find the difference in target neural recording modality alone to be sufficient grounds for considering this work novel.\n\nQUESTIONS:\n* The authors seem to be using the encoding model almost exclusively for generating the initial set of labels. In assessing the “category” defined by these images, the authors rely on the similarity of the concept vector for each image in CLIP space. However, as they have direct access to the encoding model, couldn’t they use the encoding model to yield similarity from the model? It would also be interesting to see if a similarity metric defined on the encoding model activations could better assess the rankings. \n* Instead of just generating the most activating images, could one use the encoding model to find the most distinctive set of images (e.g., images that only selectively activate the target parcel while eliciting little to no activations in other parcels)? Such a contrastive approach in semantic assignment across multiple parcels at once may help further differentiate the selectivity of each parcel.\n\nLIMITATIONS:\nAuthors identify key limitations", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel and powerful in silico paradigm for discovering and mapping the visual selectivity of cortical regions across the entire brain. The central motivation is to overcome the limitations of traditional fMRI studies, such as high cost, small stimulus sets, and experimenter bias. The authors develop a transformer-based encoder-decoder model that learns to predict fMRI activity from natural images, leveraging a cross-attention mechanism to align deep network features with responses in specific brain parcels. By training this model on the large-scale Natural Scenes Dataset (NSD), they create a \"digital twin\" of the brain, enabling large-scale virtual experiments. This allows them to search massive image datasets (ImageNet) or use generative models (BrainDIVE) to identify or create \"superstimuli\" that are predicted to maximally activate specific parcels. The work successfully replicates known selectivity for faces, discovers novel, complex selectivity for concepts like \"skateboarding\" and \"tool use\" in previously unlabeled regions, and validates that these computationally-derived hypotheses better explain held-out brain activity than traditional baselines.\n\nSTRENGTHS AND WEAKNESSES:\n**###** **Strengths**\n\n**Innovative Computational Paradigm:** The paper's core contribution is a powerful data-driven framework for discovery that moves beyond traditional hypothesis-testing. The use of in silico experimentation to generate and test hypotheses on a massive scale is a significant methodological advance for the field.\n\n**Sophisticated and Effective Architecture:** The use of a transformer-based encoder with a cross-attention mechanism is a state-of-the-art approach to brain encoding. The systematic comparison of different backbones and the finding that DINOv2 provides a strong performance baseline adds to the paper's technical rigor.\n\n**Rigorous Multi-Level Validation:** The authors effectively validate their method by first confirming its ability to identify known category-selective regions (aTL-faces, Figure 4), and then by demonstrating quantitatively that their novel hypotheses for unlabeled parcels significantly predict held-out neural data better than a baseline model (Tables 2 & 3).\n\n**Discovery of Complex and Generalizable Selectivity:** The pipeline successfully identifies parcels with intriguing, complex selectivity (e.g., \"child eating,\" Figure 1c) and, importantly, shows that some of these preferences are consistent for the same anatomical parcel across multiple subjects (e.g., the \"tool parcel,\" Figure 6), greatly strengthening the biological plausibility of the findings.\n\n### Weaknesses and Possible Improvements\n**Lack of Experimental Validation for Generated Stimuli:** A primary limitation, acknowledged by the authors, is that the \"superstimuli\" generated by BrainDIVE have not yet been tested in new fMRI experiments. While the encoder-based reranking provides confidence, showing these novel images to subjects to confirm they elicit stronger-than-normal responses would be the ultimate validation of the generative pipeline.\n\n**Potential for Dataset Bias:** The model is trained exclusively on the NSD dataset, which may introduce biases. The authors astutely note a potential \"zebra-selective\" parcel as a possible artifact of the NSD stimulus set (Appendix A.12). Future work could strengthen the generalizability of the findings by training or fine-tuning the model on multiple, diverse fMRI datasets.\n\n**Minor Clarity Issues in Figures:** Some figures could be slightly improved for readability. For instance, the caption for Figure 3 describes two distinct subplots but does not use (a) and (b) labels to explicitly link the text to the corresponding images, which could cause momentary confusion.\n\nQUESTIONS:\n### Questions/Clarifications for the Authors\n- The parcel-based approach is grounded in functional localization, yet the brain also relies heavily on distributed representations. While the paper notes that selectivity is \"graded\", could you further elaborate on how you view these highly selective parcels as nodes within wider, distributed networks? Does your framework offer a way to map the connections between these specialized nodes?\n- The results depend on the Schaefer-1000 functional parcellation. How sensitive do you believe the findings are to this specific parcellation scheme? Have you considered how using a different atlas, or perhaps subject-specific functional parcels, might alter the discovered patterns of selectivity?\n- The generated BrainDIVE images (e.g., Figure 9d, 10d) are a fascinating result of the pipeline. Based on your model, what is the predicted quantitative increase in activation these \"superstimuli\" would elicit compared to the best-performing images from ImageNet or the NSD test set?\n- While the parcel selection method (Section 3.3) is very well-reasoned, the general low predictive accuracy in many non-visual areas (Figure 3) remains a challenge. Do you believe this is primarily a limitation of using encoders trained on static 2D images to model regions that may process more dynamic or abstract information, or is it more related to inherent signal-to-noise issues in fMRI for these regions?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}