Transformer brain encoders explain human high-level visual responses
Authors: Hossein Adeli
Minni Sun
Nikolaus Kriegeskorte
Venue: NeurIPS 2025 spotlight ()
Keywords: Brain encoding
visual processing
transformers
Primary area: neuroscience_and_cognitive_science
TL;DR: We present a transformer brain encoder that achieves state of the art performance, by leveraging brain-region to image-feature cross-attention mechanism, efficiently mapping high-dimensional retinotopic features to brain areas.

Abstract: A major goal of neuroscience is to understand brain computations during visual processing in naturalistic settings. A dominant approach is to use image-computable deep neural networks trained with different task objectives as a basis for linear encoding models. However, in addition to requiring estimation of a large number of linear encoding parameters, this approach ignores the structure of the feature maps both in the brain and the models. Recently proposed alternatives  factor the linear mapping into separate sets of spatial and feature weights, thus finding static receptive fields for units, which is appropriate only for early visual areas. In this work, we employ the attention mechanism used in the transformer architecture to study how retinotopic visual features can be dynamically routed to category-selective areas in high-level visual processing. We show that this computational motif is significantly more powerful than alternative methods in predicting brain activity during natural scene viewing, across different feature basis models and modalities. We also show that this approach is inherently more interpretable as the attention-routing signals for different high-level categorical areas can be easily visualized for any input image. Given its high performance at predicting brain responses to novel images, the model deserves consideration as a candidate mechanistic model of how visual information from retinotopic maps is routed in the human brain based on the relevance of the input content to different category-selective regions. Our code is available at \href{https://github.com/Hosseinadeli/transformer_brain_encoder/}{https://github.com/Hosseinadeli/transformer\_brain\_encoder/}.

Transformer brain encoders explain human high-level visual responses
Hossein Adeli Zuckerman Mind Brain Behavior Institute Columbia University
Sun Minni
Zuckerman Mind Brain Behavior Institute
Columbia University
ha2366@columbiaa edu
ms57248columbia. edu
Nikolaus Kriegeskorte Zuckerman Mind Brain Behavior Institute Columbia University
nk2765@columbia.co edu
Abstract
A major goal of neuroscience understand brain computations during visual pro- cessing in naturalistic settings. A dominant approach is to use image-computable deep neural networks trained with different task objectives as basis for linear encoding models. However, in addition to requiring estimation of large number of linear encoding parameters, this approach ignores the structure of the feature maps both in the brain and the models. Recently proposed alternatives factor the linear mapping into separate sets of spatial and feature weights, thus finding static recep- tive fields for units, which is appropriate only for early visual areas. In this work, we employ the attention mechanism used the transformer architecture to study how retinotopic visual features can be dynamically routed to category-selectio show that this computational motif is
areas in high-level visual processing. significantly more powerful than alternative methods in predicting brain activity during natural scene viewing, across different feature basis models and modalities. We also show that this approach is inherently more interpretable as the attention- routing signals for different high-level categorical areas can be easily visualized for any input image. Given its high performance predicting brain responses novel images, the model deserves consideration as a candidate mechanistic model how visual information from retinotopic maps is routed in the human brain based the relevance of the input content to different category-selective regions. Our code is available https///ghbbb.oom//oosiaial///na
Introduction
An influential approach to study plausible neural computations in the brain is to train Deep Neural Network (DNN) models on different tasks [44] 28 and compare their learned representation to brain There has been great deal of discussion and research on best ways compare the learned representations to the ones recorded from the brain (across models and across models and brains). One main approach
activity 52 23
build encoding models- learn mapping function from one feature domain to another and measure the accuracy of the prediction in held-out sets [4 36). An alternative approach is characterize the geometry topology of the representation in each model or in the
brain and then compare them (e.g. RSA; (2)). In this work, we focus on the learned encoding functions, as believe that can give us further insight into the computations the brain.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

The visual system uses structured retinotopic maps as it processes visual information in the cortex. Not surprisingly, models, such as Convolutional and transformer neural networks, that also maintain retinotopic maps of the space perform best on different visual tasks (e.g. recognition and segmentation) and consistently outperform other models in different brain activation prediction benchmarks [46] 17]. However the retinotopic feature maps from deep networks presents typically have a very large number of units posing us with a challenge when mapped unto the responses in the brain. Linear encoding models, although theoretically the simplest choice, can become very high-dimensional in that case (the number of parameters equals the product of the number of model units and the number of units/voxels to be predicted) and require strong regularization (L2 penalty) given the size of typical neuroimaging datasets (6). To address these limitations, approaches have been proposed that learn spatial receptive fields (RF) for different units or voxels in the brain data, using which the representation is first aggregated across space and then the lower dimensional representation is linearly mapped to the brain responses 26
49 35 These models have been shown to perform on par with linear regression models despite having a fraction of the number of parameters and are also more plausible mechanisms of how information can route different units. However, they can only capture fixed routing where input to unit comes from a specific area in space regardless of the input content.
Transformer architectures have been extremely successful in many domains, including vision [2] and language [5]] Their success can be attributed to general and simple (therefore scalable) computational motif where information is routed based on the content. In these models, each token (be a representation of word in sentence patch in image) queries other tokens to find how relevant they are to updating its representation. The selective nature of this mixing has motivated naming this process "attention" in Transformers (51]. Then the new representation of this token becomes the average of the representation of all tokens, weighted by their degree of relevance (i.e. attention scores). We hypothesize that the optimal way for the routing of information from the retinotopic visual maps to category selective areas is to use the same computational motif where brain areas only attend to parts of the visual maps with the content relevant what the area is selective for (Fig. [ For example if there is face in the image, it could appear anywhere, but the FFA (fuisform face area) can learn to route only the information from the patches where the face-like stimuli are and then expand this lower dimensional representation in the area. Note that this approach is in a way a generalization of the aforementioned RF based methods going from fixed receptive fields to a dynamic content-hoe based receptive fields.
2 Related works
Predicting brain activity is an important objective, both as an engincering challenge and also as means of studying brain computations, reflected in the number of community- driven benchmarks such as Algonauts [17], Brain-score (46), and Sensorium [50]. The availability
Brain encoding models:
of large-scale neural datasets has necessitated innovation in new encoding models [2]]. Spatial- feature decomposition models have shown that considering the retinotopic maps and the receptive field organization can lead more efficient encoding models [26 49 35 45). Generalizing these approaches to high-level visual areas would require considering more dynamic routing motifs.
Self-supervised Vision Transformers: Transformers have been shown outperform convolutional and recurrent neural networks (CNNs) on a variety of visual tasks including object recognition [12). More recent studies have explored training these models on self-supervised objectives, yielding some intriguing object-centric properties [ that are not as prominent in the models trained for classification. When trained with self-distillation loss (DINO, (6) and DINOv2 (8 the attention values contain explicit information about the semantic segmentation of the foreground objects and their parts, reflecting that these models can capture object-centric representations without labels M.]. These findings show that features from these models can be a good basis for predicting neural activity in the brain. Recent work has also shown that networks trained using self-supervised contrastive losses (such as SimCLR; (9) match the predictive power of supervised models for high-level ventral- stream visual representations in the brain (7 8. These works argue for self-supervised learning methods as more plausible objective function for learning brain like visual representations.
Encoder-decoder Vision Transformers: Transformer-based encoder-decoder models provide a general framework that has achieved great performance in many domains [51) including domains
2

---

A
Brain responses
B
Image Backbone
Linear
Attention scores
Roi query
Scaled dot product
Feedforward
Feectorward
Selt Attension
Cress Attention
Positional encoding
Lineer
vertices
Roi
XN
Image tokens
Image patches
Positional encoding
Rol queries
Figure 1: A. Transformer Brain Encoder (TBEn) architecture. The input patches first encoded using a frozen backbone model. The features are then mapped using a transformer decoder to brain responses. B. The cross attention mechanism showing how learned queries for each ROI route only the relevant tokens to predict the vertices the corresponding ROI.
where one modality (e.g. image) is mapped onto another one (e.g. language) pioneering work to our approach is the DETR model [5] applied to the problem of object detection and grouping in images. The encoder in this model converts the image to rich object-centric features. The decoder uses learnable embeddings, called queries, corresponding to different potential objects, that gather information from the encoder features using cross-attention over several layers. After the decoding process, each object query can then be linearly mapped into to the category and bounding box for an object. The model is trained end-to-end and can detect many objects in one feedforward pass. We also employ this general framework here.
4
A related
3 Methods
3.1
Dataset
We run experiments the Natural Scene Dataset (NSD; [3) where the fMRI (functional magnetic resonance imaging) responses were collected from 8 subjects, each seeing up to 10,000 images. The reported results from subjects 1, 2, 5, and who completed all recording sessions. The surface-based fMRI responses across the three repetitions of each image were averaged for model training and testing. We use the train/test split that was introduced in the Algonauts benchmark [I7] where the last three sessions for each subject were held out to ensure that no test data were accessed during the model development and to make the prediction task natural as possible (predicting the future responses). All models were trained to predict the most visually responsive vertices in the brain Our analyses focused on as subset of approximately 15k vertices for each left and right hemispheres (LH and RH), shown in Figure 2A on a surface map. ROI level labels were provided for all the selected vertices based on visual and categorical properties (using auxiliary experiment; refer to 33 for details). The labels are for early visual areas ('VIv', 'V1d', 'V2v', 'V2d', 'V3v', 'V3d', and 'hV4'), body selective areas ('EBA', 'FBA-1', 'FBA-2', and 'mTL-boodes') face selective areas ('OFA', 'FFA-1', 'FFA-2', 'mTL-faces', and 'aTL-faces') ), place selective areas ('OPA', 'PPA', "RSC"), and word selective areas ('OWFA', 'VWFA-1', 'VWFA-2', "mfs-word'', and mTLL-words)..
3.2 Transformer Brain Encoder (TBEn)
We apply the general transformer mecoer-ddecoder framework to map images to fMRI responses. Figure DA shows the architecture our model. The input image first divided into patches (31 31
In IRRI, vertex refers to point on the surface mesh of the cortex used in surface-based analysis. It is analogous to "voxel" (volumetric pixel) in volumetric iMRI data, but defined in the 2D cortical surface space.

---

A
Left hemisphere
right homisphere
Transformer (rois)
troodkng Accuney
Transformer (rois)
60
bookg
D
rois
Body rois
Word rois
rois
rois
Transformer (rois) - Ridge regression
Hore
Erccidng
Froodinn senurney
mod
con
bodos
Hore
Figure 2: A. The general region of interest for highly visually responsive vertices in the back of the brain shown on different surface maps. B. Encoding accuracy (fraction of explained variance) shown for Subject 1 for all the vertices for the transformer model using ROIs for decoder queries. C. Encoding accuracy for individual ROIs and for ROI clusters based on category selectivity for the two hemispheres. D. The differences in encoding accuracy between the transformer and the ridge regression models showing that improvement in the former is driven by better prediction of higher visual areas.
in our dataset) of size 14 14 pixels. These image patches are input the backbone model which is a 12-layer vision transformer and frozen to be used as feature backbone.
The decoder uses input queries corresponding to different brain ROIs in different hemispheres to gather relevant information from the backbone outputs for predicting neural activity in each ROL. Note that these queries learnable embeddings for each ROI trained as part of the model training. We use a single-layer transformer for the decoder with one cross- aross-attention and feedforward projection operation. Figure [B shows the cross-attention process. The positional encoding is added to the image token representation to create the keys. This allows the ROI query to attend either to the location the content of the input tokens through scaled dot-product attention. The attention scores are then used to aggregate all the image tokens that are relevant to predict the brain activity in that ROL. The output decoder tokens are then mapped using single linear layer to IMRI responses of the corresponding ROL. In our implementation. decoder output for each ROI is linearly mapped vector with the size equal to the number of vertices in that hemisphere. The response is then multiplied by mask that is zero everywhere except for the vertices belonging to that ROI. This masking operation ensures that the gradient signal feeding back from the loss will only train linear mappings to the vertices of the queried ROI. The responses from different ROI readouts will then be combined using the same masks to generate the prediction for each hemisphere. The ROI queries, transformer decoder layer and the linear mappings are trained with the Adam optimizer 25 using rrean-squared-ere loss between the prediction and the ground truth fMRT activity for each image. We train and test the models separately for each subject.

---

4 Experiments
For all models including all the baselines, did 10-fold cross validation using the training set for each subject and averaged the model predictions across all folds. The model predictions were evaluated first using Pearson correlation between the predictions and the ground truth test data. The squared correlation coefficient were then divided by the noise ceiling (see [3] Methods, Noise ceiling estimation) to calculate the encoding accuracy as the fraction of the explained variance.
We present results using multiple different feature backbones namely, DINOv2 base model [88] ResNer50 [20], and CLIP large model 41]. For the DINOv2 backbone, inspired by prior work on human attention prediction [1), we did some preliminary analyses and found the patch level query representations (instead of values) to have slightly more predictive power and chose use them in all our experiment. For ResNet50, the feature maps from the last layer were extracted and reshaped to create the visual tokens comparable to transformers. For CLIP, we chose the large model to have the same image patch size (14) and transformer token dimension (768) to the DINOv2 base model. Unless otherwise stated, the features from the last layer of the backbone models are used as the input representation to the decoder.
We consider multiple different mapping functions compare to our proposed method. The Ridge regression model flattens the feature representation across space and feature dimensions and learns one linear mapping to the fMRI responses. We used a grid search to select the best ridge penalty to maximize performance on the validation data. The CLS + regression model lincarly maps only the CLS token from the transformer backbone to the vertex responses. This is a common practice in many ncuroscience studies to make the number of parameters more tractable. Another common model is to first reduce the dimensionality of the features using Principle Component Analyses (PCA) and then learn the linear mapping to the brain responses (PCA + regression).
For the spatial-feature factorized method, the model learns (H x W) spatial map and applies that to the input feature similar to the attention map in Figure The scores however are only learned for given ROI or vertex and are not dependent on the content of the image. The spatial map then aggregates the features to be linearly mapped the brain responses. The Saliency based integration method uses saliency map of the image, instead of Iearned spatial map, to integrate the tokens across space [24]. To implement this baseline, we used DeepGaze [30]. a state of the art saliency model, to generate bottom-up saliency maps for each image and then resized the maps and used the resulting attention values (weights) to combine the token representations to create a single token. A linear regression was then trained to map these compressed representations to vertex activations.
For the transformer brain encoder, we used 24 queries per hemisphere corresponding to the 24 ROIs. Note that not all ROIs were present in all the subjects, therefore we present results and figures for subjects individually. If an ROI is not mapped in a subject the decoder output not mapped to any vertices. The figures in the main text are generated using the results from subject 1, but the figures for the remaining three subjects are presented in the supplementary section A.1
Table I: Encoding accuracy using DINOv2 backbone
Encoder
Subjects
Model size (M)
S1
S2
S5
S7
Ridge regression
0.56
0.52
0.50
0.37
~90
CLS + regression
0.38
0.37
0.45
0.33
~2
PCA regression
0.52
0.47
0.46
0.34
22
Spatial-feature factorized (rois)
0.49
0.46
0.48
0.37
23
Saliency based integration
0.38
0.37
0.44
0.32
~2
Transformer (rois)
0.60
0.56
0.56
0.42
28
Table shows the encoding accuracy of the encoding models using the DINOv2 backbone. Ridge regression requires tuning larger number of parameters compared to the other approaches (all model sizes reported as multiples of millions of parameters).
The CLS token takes weighted average of all the salient image tokens to create compact repre- sentation of the image and the linear mapping of this token performs similarly to the Saliency based integration model. Both serve useful comparisons to highlight the benefit of transformer attention
5

---

versus generic feature reweighting. The PCA based model performs similarly to the Spatial-feature factorized but both perform worse than the full ridge regression model. Our model, leveraging the attention mechanism to flexibly route information 2 40 consistently outperforms all the baseline models across subjects. The important difference note is that our model allows each ROI to dynamically route the tokens that have relevant content for that ROI so in other words each learn to create their own "CLS" token dynamically based on the content of the image and the ROI selectivity.
Figure 2 shows the encoding accuracy of our model for subject for the areas of interest projected onto the cortical surface using Pycortex [T]]. Figure _ shows the encoding accuracy divided over all the individual ROIs and also clusters of ROIs. When we compare the transformer encoder to the ridge regression model (Fig. (Fig. we see that our model achieves higher encoding accuracies through better performance for categorical areas, This suggests that content based routing can be part of the brain computation for higher level visual areas.
We further tested whether the transformer based mapping requires a larger number of training images be effective. Table Shoows that the model can be trained using as little as a few hundred samples making it suitable for smaller scale experiments as well. Also encouraging is to see that our model can achieve accuracy on par with the baseline models with fraction of the training data.
Table 2: Encoding accuracy for different training set sizes
Encoder
Training set size
Subjects
S1
S2
S5
S7
Transformer (rois)
550
0.41
0.45
0.45
0.34
Transformer (rois)
1100
0.46
0.48
0.49
0.36
Transformer (rois)
2200
0.51
0.52
0.51
0.39
Transformer (rois)
4400
0.56
0.54
0.54
0.40
Transformer (rois)
8800
0.60
0.56
0.56
0.42
To examine whether our results depend on the specific choice of the transformer backbone architecture, we tested all the encoding models on the ResNet50 backbone features (a fully convolutional network). Table shows that we replicate the exact same pattern of accuracy as the DINOv2 backbone, where the transformer encoder outperforms the other two alternatives across all subjects. This shows that the transformer encoder can map differently learned features (transformer vs convolution) well to the brain data.
Table Encoding accuracy using ResNer50 backbone
Encoder
Subjects
Model size (M)
S1
S2
S5
S7
Ridge regression
0.49
0.48
0.47
0.37
~900
Spatial-feature factorized (rois)
0.42
0.42
0.43
0.33
~60
Transformer (rois)
0.52
0.50
0.50
0.38
~88
4.1 Vertex-based routing
So far the presented transformer encoding models used ROIs as units of routing. But the routing could be made more granular by learning a decoder query for each vertex where the gathered features from the decoder would be mapped linearly to the corresponding vertex value. This approach can also be applied in the spatial-feature encoding models where a spatial map is learned per vertex. Tsoos model accuracies for these two approaches using the vertex-based routing, indicating improvements for both models across all the subjects. Examining the encoding accuracy for individual ROIs (Fig. 3), we can see that the performance boost came almost entirely from early visuals areas for the transformer based model. The fact that shifting from ROI-based to vertex-bases routing does not improve encoding accuracy for higher visual areas indicates that ROIs may be the right level of routing for those regions, however the early visual areas requires more granular routing because the receptive fields of the vertices are smaller, more heterogencous, and less content dependent.

---

A
Transformer (vertices)
B Transformer (vertices) Transformer (rois)
Serocing POSPRRS
Vie
Nac
om
visued codas
pleass
words
Transformer (vertices)
Ridge regression
0.4
Crooding Accumac
Flyy
Encoding
X5
V08
nis
rrest
con
www.
wous
Karss
Figure 3: A. The encoding accuracy for subject 1 shown on the brain surface for the transformer model with vertices decoder queries. The difference in encoding accuracies going from ROIs to vertices as the decoder queries shows the improvement is almost entirely from the early visual areas. C. The vertex- based transformer model outperforms the ridge regression model for almost all the ROIs.
Comparing the vertex-based transformer model to the ridge regression model (Fig. 3B) shows that the former now outperforms the latter in almost all the ROIs.
Table 4: Encoding accuracy for different decoder queries
Encoder
Subjects
Model size (M)
S1
S2
S5
S7
Spatial-feature factorized (rois)
0.49
0.46
0.48
0.37
~23
Spatial-fcature factorized (vertices)
0.52
0.48
0.48
0.37
~51
Transformer (rois)
0.60
0.56
0.56
0.42
~28
Transformer (vertices)
0.63
0.59
0.57
0.44
~00
Transformer (vertices) backbone layers ensemble
0.65
0.62
0.59
0.45
~300
Motivated by previous encoding models of the brain having used CLIP embeddings 41 to represent images [32], we tested the different mapping functions using this feature backbone. Table shows while the performance is generally not as good as the DINOv2 backbone, it yields the same exact pattern of results. The Transformer-based models outperform other alternatives with the vertex-based routing reaching higher performance overall. Taken together with also the lower performance we saw with ResNet50 backbone, the DINOv2 features, a self-supervised trained vision transformer, deserve consideration as models of human visual brain representations.
Table 5: Encoding accuracy using CLIP vision backbone
Encoder
Subjects
Model size (M)
S1
S2
S5
S7
Ridge regression
0.51
0.48
0.47
0.38
+49
Spatial-featur factorized (rois)
0.38
0.35
0.40
0.31
~22
Spatial-featur factorized (vertices)
0.44
0.40
0.42
0.32
~00
Transformer (rois)
0.53
0.49
0.50
0.38
~28
Transformer (vertices)
0.55
0.52
0.52
0.40
~0

---

Transformer (vertices) with backbone layer ensemble
backbone layer for predicting each voxel
C
Transformer (vertices) with backbone layer ensemble Transformer (vertices)
00
on
esc
asto
pacco
Figure 4: A. Encoding accuracy of the transformer encoding model with vertex-based queries ensembled across backbone layers. B. Showing the backbone layer from which each vertex was best predicted. C. The improved performance of ensembling is almost entirely from better prediction of early visual areas.
4.2 Ensemble
A concern with using complex encoding models for neural system identification is that the non- linear mapping may obscure the differences in the underlying representations [21]. However, our results with different feature backbones show that the ones that perform better using the linear model consistently perform better using our transformer encoding model as well, just with the latter achieving higher accuracies.
To address this concern further, consider robust phenomenon shown consistently using linear encoding with convolutional neural network backbones, where the earlier layers of the network are better features for predicting the earlier visual areas [52] 19 37 23 533. We trained different transformer decoders with image tokens coming from different layers of the DINOv2 backbone. We then use softmax operation across the ensemble of models get the final prediction for each voxel. The softmax weights are based on goodness of the prediction for each model for that vertex in the validation set. Figure. 44/ shows the accuracy of the overall model on the brain surface for subject 1. The layers that had the highest weights in the ensemble for predicting for each given voxel is shown 43: higher visual areas were better predicted by later backbone layers, indicating that backbone layers capture similar feature abstractions as the brain.
Comparing the ensemble model the model trained using only the final backbone layer features (Fig. 4C), we can see that the performance increase is entirely driven by better prediction of earlier visual areas. These results show that our encoding model does obscure the differences in the underlying representation pointing further to its plausibility.
4.3 Attention maps
Different methods have been developed to interpret linear encoding models to make claims about the the selectivity learned for each ROI. Some methods tend to retrieve or generate images that highly and others focus creating importance maps to show which parts of the input images are important for predicting the activity of an ROI [43].
activate the ROI vertices K3 34
The difference in our approach is that the cross-attention scores (Fig. TB) can be examined to reveal the selectivity for each ROI, making our model inherently more interpretable. We visualize the attention maps for 3 different ROIs in Figure for the transformer encoder trained with ROI decoder queries with DINOv2 backbone. First an early visual area, V2d (dorsal) in the left hemisphere. Since the visual field is flipped around both horizontal and vertical meridians in the cortex (starting from the retina), we expect the brain activity this area to represent visual information from the bottom- tattom-right right of the input (given that the subjects were instructed to hold fixation at the center of the screen for the presentation duration). We see this exact pattern emerge in the attention maps. Recall that the decoder queries can learn to attend to both patch locations their content (since the key value the sum of backbone image patches and positional encoding). this case, the attention seems to be completely driven by the location, similarly for all the images, ignoring the content. This

---

V2d in the left hemisphere
OFA the right hemisphere
FBA the right hemisphere
Figure 5: Attention maps. Transformer decoder cross attention scores for three ROIs overlaid on the images (with brighter colors indicating higher attention weights). The selected ROIs show different ways in which the learned ROI queries can route information- based on location (V2d), content (FBA), or a combination of the two (OFA) depending on the location of the ROI in the brain processing hierarchy.
exactly what we would expect from an early visual area. The fact that all the vertices this ROI have to share the same attention map hurts accuracy as we saw in Figure DD, since the vertices do have smaller RFs this area than a quadrant, however this can addressed by vertex level routing.
The second ROI is OFA in the right hemisphere. a mid-level face selective area [16]. The attention maps is this area consistently focus on faces. Since this area is in the right hemisphere it also has a preference for visual input in the left visual field. We can see this for cases with multiple faces where the faces in the right visual field are not strongly attended. The decoder query therefore makes use of both the positional encoding and the content component of the key to attend to the most relevant part of the image to predict vertices in this ROI. The attention could also be spread across multiple faces in different locations. This is the important dynamic aspect of the receptive field in higher visual areas that can be captured using the transformer attention mechanism. The third area FBA in the right hemisphere, a high level body selective area (39]. The attention maps are more spread across bodies for this ROI and not just faces. Supplementary section A.. includes quantitative analyses of the category selectivity of the attention maps. In the Supplementary section scction[A.3] provide an analyses of the similarity between the learned queries for different ROIs (capturing visual and semantic similarity between them) and also show how model can be used in a pipeline using to generate stimuli that maximally activate different ROIs (section A.4)
diffusion models
32
Table 6: Encoding accuracy using BERT backbone
Encoder
Subjects
Model size (M)
S1
S5
S7
Ridge regression
0.19
0.21
0.25
0.19
~900
Transformer (rois)
0.27
0.27
0.33
0.27
~8
4.4
Text modality
So far we have tested the transformer encoding model on a few vision backbones but is this approach generalizable to other modalities? To test this, we first used the BLIP model [31] to generate short captions for all the images in the dataset. Using BERT [10 as the feature backbone, the decoder works exactly as before, using ROI queries to map backbone features to fMRI responses. Table shows how the transformer model outperforms the regression model across all subjects (with fraction of the parameters). Given only semantic information available in the captions, the model can only predict the high level visual areas shown in Figures 6A and 6B.

---

Transformer (rois)
Transformer (rois)
120
100
rac.
706-3
viout
Encod BOoueey
Figure 6: A. Transformer encoder accuracy using image caption as input B. Only high-level visual areas are predicted by semantic information in caption.
5 Discussion
Linear encoding models have been the dominant method used for learning the mapping from model features to brain activity [14]. The reasons for this (see [2I] for review of these points) include theoretical simplicity, allowing comparison among backbone features, biological plausibility, and the ability interpret the learned weights. However, this approach parameter inefficient for a typical number of voxels and image features, ignores the organization of the features, and does not capture nonlinear computations between brain areas such ubiquitous normalizations [48). Our proposed routing based method not only reaches state of the art accuracy, it also achieves the aforementioned desiderata for encoding models, as we have shown our results.
Foundation vision models (c.g. DONOv2 or CLIP) trained with self-supervised objectives can serve general visual representation backbones. However these task agnostic models do not capture all the computations the brain and between brain areas, which needs to be addressed by learning better encoding models. Our work suggests mechanism for how different brain areas dynamically gate their input based on the input content and the area selectivity. A flexible routing mechanism reflected in deviation from the classical RF characterization of responses, so content-dependent Our results showing that the
RF shifts provide evidence for a more flexible mechanism encoding accuracy for high-level areas cannot be improved beyond ROI-based routing also agrees with prior work on between area interactions using communication subspaces [47] which can also be modulated by attention. The routed information that relevant an area can then get expanded more in-depth. This process allows for cutting down on wiring cost in the brain by not connecting all the units one area to another area but rather only subset of relevant information getting routed with more local connections expanding the representation.
442
Limitations: We performed our experiments on NSD []. the largest image viewing fMRI dataset to date. It will be important to test the generality of our approach on other datasets using different recording techniques (Neurophysiology, EEG, etc) I3 and on different input modalities (such as video and audio). We used vertex-wing routing to capture the responses in early visual areas but while the computations for smaller receptive fields can be learned by this approach, the way the RFs are implemented in the brain are through different anatomical and wiring constraints. Also we chose for the model to read out the brain responses from backbone both early and high-level visual areas. Future work will seek to explore the connectivity between early and high-level visual areas in a more integrated system and test whether making the model further aligned with known anatomy of the visual cortex will improve performance.
10

---

Acknowledgments
Research reported in this publication was supported in part by the National Institute of Neurological Disorders and Stroke of the National Institutes of Health under award numbers IRFINS SIINS12880 128897 and 4ROINS 128897. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.
References
[1] Hossein Adeli, Seoyoung Ahn, Nikolaus Kriegeskorte, and Gregory Zelinsky. Affinity-based atten- tion in sele-supervised transformers predicts dynamics of object grouping in humans, arXiv preprint arXiv:2306. 002994 2023.
[2] Hossein Adeli, Sun Minmi, and Nikolaus Kriegeskorte. Predicting brain activity using Transformers. August 2023. URL httt::/bboorxii http://wwwwwg//ooooo/o////////// 11w.0000111111111.1111 00..2/3.88.0..517777... 551743
[3] Emily J Allen, Ghislain St- Yve, Yihan Wu, L Breedlove, Jacob S Prince. Logan T Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, et al. A massive 7t fmri datasct to bridge cognitive neuroscience and artificial intelligence. Nature neuroscience, 25(1):116-126, 2022.
[4] Mehdi Azabou, Vinam Arora, Venkataramana Ganesh, Ximeng Mao, Santosh Nachimuthu, Michael J. Mendelson, Blake Richards, Matthew G. Perich, Guillaume Lajoie, and Eva L. Dyer. A unified, scalable framework for neural population decoding, 2023. Uttpststs:///axi wwtesivvorrg//ee org/ aa///bbb/22/// 2310. 16046
Nicolas Carion, Francisco Massa, Gabriel Synnaeve, Nicolas Usunier, Alexander Kirillov, and Sergey Zagoruyko. End-to-end object detection with transformers. European conference on computer vision. pages 219-229. Springer, 2020.
[6] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pages 9650 9660, 2021.
[7] Diego Gareia Cerdas, Christina Sartzetaki, Magnus Petersen, Gemma Roig. Pascal Mettes, and Iris Groen. Brainactiv: Identifying uisuo-semantc properties driving cortical selectivity using diffusion- image manipulation. bioRxiv, pages 2024-10, 2024.
Honglin Chen, Rahul Venkatesh, Yoni Fricdman, Jiajun Wu, Joshua Tenenbaum, Daniel LK Yamins, and Daniel M Bear. Unsupervised segmentation real-world images via spelke object inference. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXIX, pages 719-735. Springer, 2022.
[9] Ting Chen, Simon Komblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning visual representations. International conference on machine learning, pages 1597-1607. PMLR, 2020.
[10] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidi- rectional transformers for language understanding, In Proceedings the 2019 conference the North American chapter of the association for computational linguistics: human language technologies, volume (long and short papers), pages 4171-4186, 2019.
[11] Arna Dhar.
face-detection. 2023. Hugging Face repository.
URL https://hugginggaace http://aaac.ccccaaaaaaaaae
[12] Alexey Dasovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenhorn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv preprint arXiv:2010.11929 2020.
[13] Matteo Ferrante, Tommaso Boccato, Grigorii Rashkov, and Nicola Toschi. Towards neural foundation models for vision: Aligning eeg, meg, and fmri representations for decoding, encoding, and modality conversion. arXiv preprint 0rXiv:2411.00972 2024.
[14] Jack Gallant, Shinji Nishimoto, Thomas Naselaris, and MC Wu. System identification, encoding models, and decoding models: a powerful new approach to fmmi research. Visual population codes: Toward common multivariate framework for cell recording and functional imaging, pages 163-188. 2012.
11

---

[15] James S. Gao, Alexander G. Huth, Mark D. Lescroart, and Jack L. Gallant. Pycortex: an interactive surface visualizer for IMRL. Fromtiers in Neuroinformatiic, September 2015. ISSN 1662-5196. doi: 10. 3389/mnin1.2115.00022
3tt.://nn...9///nnnn. 2015
a0023/abstract
[16] Isabel Gauthier, Michael J Tarr, Jill Moylan, Pawel Skudlarski, John C Gore, and Adam W Anderson. The fusiform "face area" is part of a network that processes faces at the individual level. Journal of cognitive neuroscience, 12(3):495-53) 504, 2000.
[17] Alessandro Gifford, Benjamin Lahner, Sari Saba-Sadiya, Martina G Vilas, Alex Lascelles, Audo Oliva, Kendrick Kay, Gemma Roig, and Radoslaw M Cichy. The algonauts project 2023 challenge: How human brain makes sense of natural scenes. arXiv preprint arXiv:2301.03 2023.
[18] Charles D. Gilbert and Wu Li. Top-down influences on visual processing. Nature reviews neuroscience, 14 (5):350-333, 2013. ISSN 1471-003X.
[19] Umut Güşlü and Marcel AJ Van Gerven. Deep neural networks reveal gradient the complexity of neural representations across the ventral stream. Journal of Neuroscience, 35(27):10005-100 10014, 2015.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pottern recognition, pages 770-778, 2016.
[21] Anna A Ivanova, Martin Schrimpf, Stefano Anzellotti, Noga Zaslavsky, Evelina Fedorenko, and Leyla Isik. Beyond linear regression: mapping models in cognitive neuroscience should align with research goals. Neurons, Behavior, Data analysis, and Theory, 1, 2022.
[22] Glenn Jocher. Ultralytics yolov5, 2020.
[23] Seyed-Mahdi Khaligh-Ratt Razavi and Nikolaus Kriegeskorte. Deep supervised. not unsupervised, models may explain it cortical representation. PLoS computational biology, 10(1 10111))e003391, 2014.
[24] Meenakshi Khosla, Gia Ngo, Keith Jamison, Amy Kuceyeski, and Mert Sabuncu. Neural encoding with visual attention. Advances in Neural Information Processing Syxtems, 13:15942-15553, 2020.
[25] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv: /412.698)) 2014.
[26] David Klindt, Alexander Ecker, Thomas Euler, and Matthias Bethge. Neural system identification for large populations separating "what" and "where". Advances in neural information processing systems, 30, 2017.
[27] Talia Konkle and George A Alvarez. A self-supervised domain-general learning framework for human ventral stream representation. Nature communications, 13(1):491, 2022.
[28] Nikolaus Kriegeskorte. Deep neural networks: a new framework for modeling biological vision and brain information processing. Annual Review of Vision Science, 1:417-446, 2015. ISSN 2374-4642.
[29] Nikolaus Kriegeskorte, Maricke Mur, and Peter Bandettini. Representational similarity analysis- connecting the branches of systems neuroscience. Frontiers in systems neuroscience, 2:249, 2008.
[30] Matthias Kümmerer, Lucas Theis, and Matthias Bethge. Deep gaze i: Boosting saliency prediction with feature maps trained imagenet. arXiv preprint 1rX4v 2014.
[31] Junnan Li, Dongxu Li, Caiming Xiong, and Steven Hoi. Blip: Bootstrapping tanguage-image pre-training unified languag understanding and generation. In International conference on machine learning. pages 12888-12900. PMLR, 2022.
[32] Andrew Luo, Maggie Henderson, Leila Wehhe, and Michael Tarr. Brain diffusion for visual exploration: Cortical discovery using large scale generative models. Advances in Neural Information Processing Systems, 36:75740-7578. 2023.
[33] Andrew Luo, Margaret M Henderson, Michael J Tarr, and Leila Wehbe. Brainscuba: fine-grained natural language captions of visual cortex selectivity, arXiv preprint arXiv:2310.04420, 2023.
[34] Andrew F Luo, Jacob Yeung, Rushikesh Zawar, Shaurya Dewan, Margaret M Henderson, Leila Wehbe, and Michael Tar. Brain mapping with dense features: Grounding cortical semantic selectivity in natural images with vision transformers. arXiv preprint 4rXiv:2410.05266, 2024.
12

---

[35]
Konntantin-Klemens Lurz. Mohammad Bashiri, Konstantin Willeke, Akshay K Jagadish, Eric Wang. Edgar Y Walker, Santiago A Cadena, Taliah Muhammad, Erick Cobos, Andreas S Tolias, et al. Generaliza- tion in data-driven models of primary visual cortex. BioRxiv, pages 2020-10, 2020.
[36] Thomas Nasclaris, Kendrick N Kay, Shinji Nishimoto, and Jack L Gallant. Encoding and decoding in fmri. Newroimage, 56(2):400-410, 201
[37] Aran Nayebi, Javier Sagastuy- Beggattuy-Brena, Daniel M Bear, Kohitij Kar, Jonas Kubilius, Surya Ganguli, David Sussillo, James J DiCarlo, and Daniel LK Yamins. Goal-driven recurrent neural network models of the ventral visual stream. bioRxiv, 2021.
[38]
Maxime Oquab, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaacidin El-Nouby, Russell Howes, Po- Yao Huang, Hu Xu, Vasu Sharma, Shang-Woc Wen Li, Wojeiech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Synnaeve, Ishan Misra, Herve Jegon, Julien Mairal, Patrick Labatut, Armand Joulin, and Piotr Bojanowski. Dinov2: Learning robust visual features without supervision, 2023.
[39] Marius V Peelen and Paul E Downing. Selectivity for the human body in the fusiform gyrus. Journal of neurophysiology. 93(1):603-608, 2005.
[40] Pawel A Pierzchlewicz, Konstantin F Willeke, Arne F Nix, Pavithra Elumalai, Kelli Restivo, Tori Shinn, Cate Nealley, Gabrielle Rodriguez, Saumil Patel, Katrin Franke, al. Energy guided diffusion for generating neurally exciting images. In Proceedings of the 37th International Conference on Neural Information Processing Systems, pages 32574-32601, 2023.
[41] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pages 8748 8763. PmLR, 2021.
[42] Nirmala Ramalingam, Justin NJ McManus, Wu Li, and Charles D Gilbert. Tow-down modulation of lateral interactions in visual cortex. Journal of Neuroscience, 33(5):1773-1789, 2013.
[43] N Apurva Ratan Murty, Pouya Bashivan, Alex Abate, James J DiCarlo, and Nancy Kanwisher. Compu- tational models of category-selective brain regions enable high-throughput tests of selectivity. Nature communications, 12(1):5540, 2021.
[44] Blake A Richards, Timothy P Lillicrap, Philippe Beaudoin, Yoshua Bengio, Rafal Bogacz, Amelia Christensen, Claudia Clopath, Rui Ponte Costa, Archy de Berker, Surya Ganguli, et al. A deep learning framework for neuroscience. Nature neuroscience, 22( (2(11):161-1170) 2019.
[45] Shreya Saha, Ishaan Chadha, et al. Modeling the human visual system: Comparative insights from response-opiimize and task-optimized vision models, language models, and different readout mechanisms. arXiv preprint arXiv:2410. 14031, 2024.
[46] Martin Schrimpf. Jonas Kubilius, Ha Hong. Najib Majaj. Rishi Rajalingham, Elias B Issa, Kohitij Kar, Pouya Bashivan, Jonathan Resy Franziska Geiger, et al. score: Which artificial neural network for object recognition is most brain-like? BioRxiv, page 407007, 2018.
[47] João D Semedo, Amin Zandvakili, Christian K Machens, Byron M Yu, and Adam Kohn. Cortical areas interact through communication subspace. Newron, 102(1):2492259, 2019.
[48] H Sebastian Seung. Interneuron diversity and normalization specificity in visual system. bioRxiv, pages 2024 04, 2024.
[49] Ghislain Ste and Thomas Naselaris. The feature- feature-weighted receptive field: an interpretable encoding model for complex feature spaces. Neurolmage, 180: 188-202, 2018.
[50] Polina Turishcheva, Paul G Fahey, Michaela Vystrčilová, Laura Hansel, Rachel Froebe, Kayla Ponder, Yongrong Qiu, Konstantin F Willeke, Mohammad Bashiri, Eric Wang, al. The dynamic sensorium competition for predicting large-scale mouse visual cortex activity from videos. ArXiv, pages arXiv-2305, 2024.
[51] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones. Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. arXiv preprint arXiv:1706.088
2017. [52] Daniel LK Yamins, Ha Hong, Charles Cadieu, Ethan A Solomon, Darren Seihert, and James DiCarlo. Performance-optimizzd hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, (11(23)88619 8624, 2014.
[53] Huzheng Yang, James Gee, and Jianbo Shi. Brain decodes deep nets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2000-230400 2024.
13

---

A
Supplementary Material
A.1 Encoding accuracies for Subjects 2, and 7
Transformer (rois)
Subject 2
1.0
Left Hemisphere
Encoding Accuracy
Right hemisphere
0.6
0.2
0.0
VIV
V1d
V2
V2d
V3v
Vad
EBA
FBA
OFA
PRA-
FFA-2
OPA
PP4
RSC
OWIF
vWWF
WVFA-2
mife mof-wwordd
visuais
bodles
faces
places
All areas
ROIS
Subject 5
Hemrsphere Right Hamisphere
Encoding Accuracy
0.6
0.2
V1v
V2a
V2v
V2d
vBv
V3d
hv4
EBA
FBA-1
FBA-2
OFA
PFA-1
PPA-2
оP4
PP4
RSC
OWFA
VWFA-1
VWFA-2
mifs words
visuals
badies
faces
es
words
All aress
Subject 7
1.0
Laft Hemisphere Right Homisphare
0.6
0.2
V1
V2d
V2v
V2d
V3v
Vad
hv4
EBA
FBA 2
OFA
FRA-1
FFA-2
OPA
PPA
RSC
OWPA
ww96-1
VWRA-2
mfs ifs-words
mIT nTTL-words
visuals
bodies
faces
places
words
All
Figure S1: Encoding accuracy (fraction of explained variance) shown for Subjects 2, 5, and 7 for individual ROIs and for ROI clusters for the two hemispheres. The transformer model uses ROIs for decoder queries and features from the last layer of the DINOv2 backbone.
14

---

Transformer (rois) Ridge Regression
Subject 2
Left Hemisptere
0.2
0.0
-0.
Vtd
V2d
vav
v3d
hvd
EBA
FRA 1
FRA-2
Osa
FRA-2
FFA-2
n9A
RSC
OWF4
WwF-1
VWFFA-2
nás words
visuals
bodies
facos
places
words
All
ROIs
Subject 5
0.4
Hemispherc Right Hemisphere
Encoding
0.0
-02
VIv
v2d
v2d
v3v
vad
hV4
EBA
FBA-1
FBA-2
OFA
FR4-1
FFA-2
OPA
PPA
RSC
OWFA
VWFA-1
VWFF-2
mis-words
visuals
bodies
faces
places
words
All areas
ROIs
Subject 7
0.4
Left Hemisphere
Right Hemisphere
0.0
VIv
Vdd
V2v
V2d
vav
V3d
hv4
EBA
FBA-2
or4
FFA-1
FRA-2
OPA
PPA
RSC
OWFA
WVFA-1
WWFA-2
mifs words
ITLwwords
visuats
bodies
faces
placos
words
All areas
Figure S2: The differences in encoding accuracy between the transformer and the ridge regression models shows that the transformer encoder better predicts especially higher visual areas.
15

---

Transformer (vertices)
Subject 2
1.0
Left Hemisphere
Right hemisphree
0.6
Encoding
0.2
0.0
VIa
V2v
V3v
V3d
AV4
EB4
FBA
OF4
FPA.1
FFA-2
PPA
RSc
OWFA
VWFA
VWFA-2
iif-wwords
vinuats
bodies
faces
places
words
areas
Subject 5
Hemisphere Right Hemisphere
Encoding Accuracy
0.6
0.2
V1v
V2d
V2v
V2d
V3v
v3d
hv4
ESA
FBA-1
FBA-2
OFA
PRA-1 1
PFA-2
OPA
PPA
ASC
OWFA
VWFFA-
WWFA-2
mís words
visuols
bodies
faces
words
All
ROIS
Subject 7
Encoding Accuracy
0.8
Right Homisphere
0.4
0.0
V2V
V2d
V2v
V2
V3v
V3d
hv4
EBA
FB4-2
OFA
FFA-1
FRA-2
OPA
PPA
ASC
OWF4
vWFA-11
VWFA-2
mfs- words
mTT L-words
visuals
bodies
faces
places
words
Ali areds
ROIs
Figure S3: Encoding accuracy (fraction of explained variance) shown for Subjects 2, 5, and 7 for individual ROIs and for ROI clusters for the two hemispheres. The transformer model uses vertices for decoder queries and features from the last layer of the DINOv2 backbone.
16

---

Transformer (vertices) Transformer (rois)
Subject 2
0.4
Left hemishere
Right Hemisphere
0.2
Encoding
0.0
-0.
Viu
V2d
V2v
V2d
vav
hv4
EBA
FBA-2
OFA
FRA-
FPA-2
OPA
PPA
ASC
OWFA
VWFFA-1
UWFA-2
mis mis-wordss
visuals
bodies
faces
places
words
areas
ROIs
Subject 5
0.4
Left Hemisoherc Hemitihee
Enceding
0.0
-.2
Vid
V2d
V3d
hv4
EBA
FBA FBA-1
FBA-2
OFA
FRA-J
FFA-2
OPA
PPA
RSC
OWPA
VWFF-1
VWFFA-2
mfs-words
vicuels
bodies
faces
places
words
All areas
Subject 7
Right Hemisphere
0.2
Encoding
-.2
V1v
Vid
V2v
V2d
V3d
hy4
EBA
FBA-
OFA
FFA-1
FRA-2
OPA
RSC
OWFA
VWPA-1
VWFFA-
mit mif-words
inToowords
vicuels
bodies
faces
places
words
areas
ROIs
Figure S4: The differences in encoding accuracy between the transformer model using vertices and the model using ROIs as decoder queries. The figure shows that any potential improvement in the former is driven by better prediction of early visual areas.
17

---

A.2 Category selectivity of attention maps
To quantify the calegory selectivity of attention maps, we classified each pixel of the test set images using YOLOv5 [2 and YOLOv8-face [T] into five categories: background, face, body (pixels classified person but face), animal, and food. each ROI, we calculated and resized its attention maps to 434 x 434 for these images, and reported the categories of the 2k pixels with top attention values. We found that the category selectivity is consistent with ROI labels, with EBA most selective for body, FFA most selective for face, and OPA/PPA/RSC most selective for hackground.
Table 7: Category selectivity of ROI attention for subject
background
face
body
animal
food
EBA
0.03
0.36
0.61
0.00
0.00
FFA-1
0.00
0.79
0.16
0.05
0.00
FFA-2
0.00
0.83
0.17
0.00
0,00
OPA
0.54
0.12
0.14
0.05
0.15
PPA
0.44
0.25
0.11
0.10
0.10
RSC
0.66
0.23
0.11
0.00
0.00
Table Category selectivity of ROI attention for subject
background
face
body
animal
food
EBA
0.03
0.25
0.72
0.00
0.00
FFA-1
0.05
0.57
0.19
0.18
0.00
FFA-2
0.05
0.53
0.27
0.15
0.00
OPA
0.74
0.15
0.06
0.05
0.00
PPA
0.78
0.11
0.08
0.03
0.00
RSC
0.71
0.19
0.09
0.00
0.00
Table 9: Category selectivity of ROI attention for subject
background
face
body
animal
food
EBA
0.29
0.26
0.40
0.05
0.00
FFA-1
0.12
0.67
0.13
0.08
0.00
FFA-2
0.10
0.59
0.28
0.03
0.00
OPA
0.31
0.33
0.20
0.16
0.00
PPA
0.36
0.29
0.20
0.15
0.00
RSC
0.08
0.37
0.50
0.00
0.06
18

---

Table 10: Category selectivity of ROI attention for subject 7.
background
face
body
animal
food
EBA
0.38
0.05
0.43
0.09
0.05
FFA-1
0.00
0.88
0.02
0.10
0.00
FFA-2
0.17
0.20
0.38
0.20
0.05
OPA
0.40
0.18
0.14
0.28
0.00
PPA
0.51
0.29
0.16
0.05
0.00
RSC
0.56
0.25
0.17
0.02
0.00
A.3 Analyzing learned ROI queries
We analyzed the representational similarity of learned ROI queries, report the average cosine similarity between each pair ROIs across 20 models trained using five different random seeds and four different DINOV2 backbone layers in Figures S5 S6 S7 S8 These figures show the visual and semantic similarity between the ROIS as reflected in the learned queries for the subjects. We observed that ROIs with shared category selectivity form clusters (faces, places, bodies, words) in the similarity matrix, exhibiting greater representational similarity within each category type.
We also see
caneegeggiiacll areas. Additionally, ROIs within the ventral early visual areas (VIv, V2v. V3v) are more similar to one another than to their dorsal counterparts (VId, V2d, V3d). and vice versa (the checkerboard patterns), reflecting the anatomical and functional organization of the visual cortex, and that the attention will be mostly driven spatial information.
divide between categorical
19

---

subject 1
Ih-Viv
1h-V1d
Ih-V2v
Ih-V3v
Ih-hV4
1.0
Ih-FBA-1
In-OFA
1-FFA-1
Ih-OPA
Ih-PPA
Ih-R5C
Ih-OWFA
Ih-VWFA-1
In-VWFA-2
In-mfs-words
m-Vid
0.4
m-V2d
m-V3d
0.2
rn-EBA
rh-FBA-1
fh-FBA-2
0.0
M-OFA
mh-FFA-1
M-FFA-2
hh-OPA
rh-PPA
rh-RSC
rh-OWFA
rh-VWFA-2
I--Iv
In-Vid
Ih-22v
In-V2d
In-33v
In-V3d
Ih-V44
In-EBA
Ih-FBA-1
Ih-OFA
Ih-FFA-1
In-09A
Ih PPA
Ih-RSC
Ih OWFA
In-VWFF-11
In-VWFA-2
Ih-mis-words
rh-V1v
m--1d
rh-V2v
ro-V2d
rh-v3v
ma-/3d
m--V44
MhEBB
hh-BBA-1
R--BA-22
rh-OFA
th-FFA-1
rh-FFA-2
M-OPA
m-PPA
M-RSC
mh-OWFA
mh-VFA-1
m-VVFF-22
Figure S5: Cosine similarity between learned ROI queries for subject 1. Each entry in the matrix represents the average cosine similarity between the query for the ROI indicated by the row label and that indicated by the column label. ROIs from the left hemisphere are labeled with 'lh', and those from the right hemisphere with 'th'. Results averaged across 20 models, trained using five random seeds and four different backbone layers.
20

---

subject 2
Ih-V1v
Ih-V2v
Ih-V2d
Ih-V3v
Ih-V3d
Ih-EBA
1.0
Ih-FBA-2
1h-OFA
1h-FFA-1
1h-OP OPA
Ih-PAA
Ih-R5C
Ih. Oh-O
0.6
Ih-VWFA-2
th-mfs-words
rh-V1v
rh-Vid
0.4
rh-V2d
rh-V3v
rh-V3d
0.2
rh-hV4
fh-FBA-2
rh-OFA
0.0
rh-FFA-1
rh-FFA-2
rh-PPA
fh-R5C
rh-OWFA
rh-VWFA-1
rh-vWFA-2
rh-mfs-words
Ih-VIv
Ih-vid
Ih-22v
Ih--2d
Ih-/3v
Ih-v3d
Ih-h44
In-EBA
Ih. FB-FBA22
In-OFA
Ih-FFA-1
Ih-FFA-2
Ii-0PA
Ih-PPA
IN-R5C
Ih- nn-VFFA-1
1haVVFA-2
Ih- mimis: words
<h-/vv
m--Vd
rh-22v
m--V2
rh-33v
m--33
M-EBA
rh-OFA
rh-FFA-1
1h-FFA-2
Th-PPA
M-RSC
Ih-OFFA
n-VVFA-1
m--VFA-2
Figure S6: Cosine similarity between learned ROI queries for subject
21

---

subject 5
Ih-VIv
hh-11d
1h-V2v
hh-V2d
Ih-v3v
Ih-hV4
Ih-EBA
1.0
1h-FBA-1
Ih-OFA
1h-fFA-1
1h-FFA-2
0.8
Ih-OPA
Ih-PPA
Ih-R5C
Ih-VWFA-1
0.6
Ih-VWFA-2
In-mfs-words
rh-V1d
rh-V2v
0.4
rh-V3v
rh-/3d
rh-hV4
rh-FBA-1
rh-FBA-2
rh-FFA-1
rh-FFA-2
Ih-OPA
Rh-R5C
rh-OWFA
1h-VWFA-1
rh-VWFA-2
th-mfs-words
Ih-Vld
Ih-22v
ln-V2d
In-33v
Ih-v3d
Ih-1v4
Ih. -BA-
Ih. F
Ih-OFA
Ih-FFA-1
Ih-FFA-2
In-OPP
Ih-PPA
In-RSC
Ih. OWFA
I--VEA-
Ih-VWAA-2
Ih-mis-wee n-mmf--wods
rh-/1v
m--Vd
rh-V2v
m--V2
rh.33v
rh-hV4
Mh- EBA
th-FBA-1
11-FBA-2
Ih-OFA
rh-FFA-1
1h-FPA-2
Mh-SC
FF-OWFA
mh-VVFA-1
m-VVFA-2
mh-
Figure S7: Cosine similarity between learned ROI queries for subject
22

---

subject 7
Ih-V1v
Ih-Vld
1h-V2v
Ih-22d
Ih-V3v
Ih-V3d
Ih-hV4
Ih-EBA
1.0
Ih-FBA-2
Ih-OFA
Ih- 1h-FFA-1
1h-F
0.8
Ih-OPA
Ih-PPA
Ih-OWFA
Ih-VWFA-1
Ih-2
lh-mfs-words
m--1v
rh-Vid
0.4
rh-V2v
rh-V2d
rh-V3d
0.2
rh-hV4
rh-FBA-2
rh-OFA
0.0
rh-FFA-2
rh-OPA
fh-R5C
rh-OWFA
rh-VWFA-1
th-VWFA-2
n-mms-words
Ih~Vvv
Ih-yld
In-V2v
Ih-V2d
Ihw33v
Ih-v3d
Ih-AV4
In-EBA
Ih-FBA-2
Ih-OFA
Ih-FFA-1
In-FFA-2
Ih-OPA
Ih,PPA
Ih-R5C
h-OWFA
Ih-VWFA-1
In-VNIFA-2
Ih-mis- Ih-mil-words words
7h1Viv
rh-Vld
rh-42v
rh-22d
rhv3v
mh-v3
rh-hV4
MTEBBA
th-FBA-2
<h-OFA
7h-FFA-2
m--OPA
mhPPA
m-RSC
INOWFA
m-VNIFA-
m-W-FAA 2
words
rh-mfs-
Figure S8: Cosine similarity between learned ROI queries for subject
A.4 Generating maximally activating images for ROIs
BrainDiVE 32 is generative framework for synthesizing images predicted to activate specific regions of the human visual cortex. It guides the denoising steps of diffusion model using gradients derived from brain encoding model. Given the strong performance of encoding model in predicting brain activity, tested whether could also effectively guide image generation within BrainDiVE framework, images optimized to maximally activate the average predicted response a specific ROI cluster, and display the top five in Figure 59 S10 The categories of the generated images are consistent with the reported category selectivity of each ROI cluster in the literature.
generated 200
23

---

subject 1: Body selective areas (EBA, FBA-1, FBA-2)
subject 1: Face selective areas (OFA, FFA-1, FFA-2)
subject 1: Place selective areas (OPA, PPA, RSC)
subject 1: Word selective areas (OWFA, VWFA-1, VWFA-2)
SLEND
subject 2: Body selective areas (EBA, FBA-1, FBA-2)
subject 2: Face selective areas (OFA, FFA-1, FFA-2)
subject Place selective areas (OPA, PPA, RSC)
subject 2: Word selective areas (OWFA, VWFA-1, VWFA-2)
Okon
Xo)
ToA.y
Figure 89: Images generated to maximally activate different ROI clusters for subjects and Using our encoding model within the BrainDiVE framework, we generated 200 images predicted to maximally activate a specific ROI cluster for a given subject (indicated by the row titles). For each cluster, we display the top five images with the highest predicted activation, as determined by our encoding model.
24

---

subject 5: Body selective areas (EBA, FBA-1, FBA-2)
subject 5: Face selective areas (OFA, FFA-1, FFA-2)
subject 5: Place selective areas (OPA, PPA, RSC)
subject Word selective areas (OWFA, VWFA-1, VWFAA-)
OC
DuIe GA
subject 7: Body selective areas (EBA, FBA-1, FBA-2)
subject 7: Face selective areas (OFA, FFA-1, FFA-2)
subject Place selective areas (OPA, PPA, RSC)
subject 7: Word selective areas (OWFA, VWFA-1, VWFA--)
Figure S10: Images generated to maximally activate different ROI clusters for subjects 5 and
25

---

B
Compute used
We used GPUs (NVIDIA L40s), memory, and storage resources from internal cluster. Storage for the entire project totals roughly 3TB. Training the model used roughly 4,000 GPU hours. Running the remaining experiments used roughly 1,000 GPU hours. The full project required more compute than these estimates due failed experiments, experiments not included the paper, and model iteration.
26

---

NeurIPS Paper Checklist
1. Claims
Question: the main claims made the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
All our claims directly backed by presented results.
Guidelines:
The answer NA means that abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match thcorctical and experimental results, and reflect how much the results can be expected to generalize other settings.
is fine to include aspirational goals as motivation long it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
We discuss the limitations on page 9.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed the paper.
The authors are encouraged create separate "Limitations" scction their paper.
The paper should point out any strong assumptions and how robust the results these assumptions (e.g.. independence assumptions, noiseless settings, model well well-specifieant specification. asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
violations of
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend implicit assumptions, which should he articulated.
The authors should reflect the factors that inffuence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or speech-to text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Our paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross-referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems.
27

---

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by
formal proofs provided in appendix or supplemental material.
Theorems and Lemmas the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: We include all the experimental details.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible important, regardless of whether the code and data are provided or not.
the contribution is dataset and/or model, the authors should describe steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access the model. In general. releasing code and data often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g., in the case of a large language model), releasing of model checkpoint, or other means that appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submissions provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g.. a large language model), then there should either be way access this model for reproducing the results way to reproduce the model (e.g.. with an open-source dataset or instructions for how to construct
dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the of cosed-source models, it may be that access to the model is limited some way (e.g.. to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open faithfully reproduce the main experimental results, described supplemental material? the data and code, with sufficient instructions
Answer: [Yes]
Justification:
We will upload the code as part of the supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (tttps::/nips.cc///nle
guides/CodeSubees ssiopoolicii for more details.
While we encourage the release of code and data, we understand that this might not be possihle,
so "Na" an acceptable answer. Papers cannot rejected simply for not including code, unless this is central to the contribution (e.g.. for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https: : :/nnis///.////////// cc/public/ qu..dess/ooteSuutnlesssoooooo... for more details.
28

---

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reprodueible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary understand results?
Answer: [Yes]
Justification:
We do provide the training/testing details.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary appreciate the results and make of them.
The full details can be provided either with the code, in appendix. supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate informa- tion about the statistical significance of the experiments?
Answer: [NA]
Justification: We provide the results for all the subjects in dataset separately. The reason as deseribed in the text, is that different subjects have slightly different number of vertices and ROI boundaries, so we felt was more appropriate to not average over subjects and to just transparently report the data for all of them.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by error bars, confidence intervals, statistical significance tests, least for the experiments that support the main claims the paper.
The factors variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should he explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report I-sigma error hars, but one should state it. The authors should preferably report a 2-sigma error har than state that they have 96% CI, if the hypothesis of Normality of errors not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures
symmetric error bars that would yield results that are out of range (e.g. negative error rates).
crror bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
Experiments compute resources
Question: For each experiment, does the paper provide sufficient information the computer resources (type of compute workers, memory, time execution) needed reproduce the experiments?
Answer: [Yes]
Justification: We are including all the details about computing resources in the supplementary files.
Guidelines:
The answer NA means that the paper does not include experiments.
29

---

The paper should indicate the type of compute workers CPU or GPU, internal cluster. or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics httts::/neurttie http:///rrr.ipaccpppapaee//////a httppplllc//caaaauieeeeleelaeeee
Answer: answerYes
Justification:
Our work conforms with the code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
the authors answer No, they should explain special circumstances that require deviation from the Code of Ethics.
The authors should make sure preserve anonymity (e.g.. if there is special consideration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification:
We discuss the societal impacts of our work the supplementary section.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deploy- of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is a direct path to any negative applications. the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it not needed point out that generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative socictal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release models, providing defenses in addition to attacks, mechanisms for monitor- ing misuse, mechanisms to monitor how system learns from foodback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible relcase of data or models that have high risk for misuse (e.g., pretrained language models, image generators, scraped datasets)?
Question:
Answer: [NA]
Justification: We used publicly available datasets. The encoding models do not have high risk of misuse to knowledge.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines restrictions to access the model or implementing safety filters.
30

---

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (c.g., code, data, models), used in the paper, properly eredited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: used publicly available fMRI datasets that we credit. We own the other assets.
Guidelines:
The answer NA means that the paper does not existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset. For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets released, the license, copyright information, and terms of in the package should be provided. For popular datasets, paperswi thcode. com/ datasetsaetss has curated licenses for some datasets. Their licensing guide can help determine the license a dataset.
existing datasets that are re-packaged, both original license and license of derived asset (if it has changed) should he provided.
this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: All the assets well documented and the documentation is provided.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the datasct/code/model as part of their sub- missions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and sereenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: We use publicly available datasets and the original papers can be consulted for all the details. We provide the relevant information in our paper.
Guidelincs:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethics, workers involved in data collection, curation, other labor should be paid at least minimum wage in country of data collector.
15. Institutional review board (IRB) approvals equivalent for research with human subjects
31

---

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on requirements of your country or institution) were obtained?
Answer: [NA]
Justification: used public available datasets.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country which research is conducted, approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applica- blc), such the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe usage of LLMs it important, original, or non non-standard component
core methods this research? Note that the LLM used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: We did not use LLMs in any significant way.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs any important, original, non-standard components.
Please refer LLM policy should should be described.
for what
32


Reviews:
[{"review_id": "x3xBJkUvMz", "reviewer": "Reviewer_dqys", "summary": "This paper proposes and investigates the use of transformer encoder-decoder models in predicting fMRI responses of subjects seeing images from the Natural Scene Dataset. These transformer models outperform linear mapping/ridge-regression based on transformer-features for several different \"feature backbones.\" The authors also find higher performance for vertex-based routing vs. ROI-based routing, interpretable attention maps, and some generalization to text (BERT backbone).", "questions": "1) Is there a simple way to understand the significance of an encoding accuracy between for example 0.6 and 0.8, as seen in Figure 2C? For example, one way to get at this qualitatively could be by evaluating quality of decoding based on the given encoding model. It would also be helpful to state what the best explained variance one might reasonably expect on this data (e.g., due to noise).\n\n2) Can the authors provide a definition of vertex (ideally upon first use in Section 4.1)?\n\n3) The authors write, \"Our approach proposes a mechanistic model of how visual information from retinotopic maps can be routed based on the relevance of the input content to different category-selective regions.\" Could they comment on how this model could be related to experiments in systems neuroscience, or models in computational neuroscience such as dynamic routing?\n\n3) How does the performance of transformer-based methods scale with the amount of data used?\n\n4) How much of an impact do parameters like temperature impact encoding accuracy?\n\nMinor comments: \n1) possible missing ref on Line 134 \n2) For Figures 3b/4c, though the caption makes clearer that the dash should be read as \"minus\", it would be nice to make this clearer from the subtitle itself.\n3) Is it possible to add some context/colorbar for the attention maps in Figure 5?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "I'm happy to recommend this work for acceptance. The authors are working on a highly relevant problem in neuroscience and propose a simple yet powerful alternative to the \"standard\" linear encoding approaches to predicting brain activity. The main claims of the paper are supported by experiments from multiple encoding \"backbones\". The explanations and motivations for different design choices is also well explained in the paper.\n\nI think the work could benefit further by explaining how to translate higher predictive performance into qualitative insights for particular ROIs or voxels. I recognize the supplement (Figure S9) is making an attempt at this, but I think it would also be helpful to give insights into why the transformer is performing better than linear regression in certain cases. Another idea is to go further into detail into the improved interpretability relative to other approaches.", "was_revised": true, "final_justification": "I believe that this paper has a strong impact for an active research area in computational neuroscience, and the results adequately support the proposed approach. The reviewers have adequately addressed my questions and their followup experiments were insightful.", "rebuttal": "We thank the reviewer for their very thoughtful comments and feedback! Please see below for our responses. \n\n> **1)** Insights from better performance\n\nWe agree that it is very important to translate better accuracy into insights about brain computation. The big insight in our work is that the encoding performance reflects whether the routing in the model is capturing the routing of information in the brain (i.e. receptive field dynamics). For the different results we highlighted this insight to give better intuitions but will do so more in the revision (e.g. an insight highlighted by reviewer EjBm: \"ROI-level granularity: Applying the same dynamic routing across all voxels in an ROI may obscure within-ROI variability—particularly in early visual areas with heterogeneous RFs. This might explain the limited gains of vertex-based queries in higher early visual regions.\").\n\n> **2)** Improved interpretability relative to other approaches.\n\nThank you for this suggestion. We showed attention maps in Fig. 5 and discussed how the attention behavior can be interpreted to reveal the selectivity of an ROI. \n\nWe have now run experiments quantifying this process. We classified each pixel of the test set images into five categories using YOLOv5 (Ultralytics., 2020) and YOLOv8-face ( Dhar, A., 2023): background, face, body, animal and food. For each ROI, we calculated and resized the attention maps to $434 * 434$ for these images, and report the categories of the$~2k$ pixels with top attention values.  We found that the category selectivity is consistent with ROI labels, with EBA most selective for body, FFA most selective for face, and OPA/PPA/RSC most selective for background. \n\nBelow shows result for subject 1, we have observed similar results with subject 2, 5, 7 and will include them in the revision.\n\n|       | background | face | body | animal | food |\n|-------|------------|------|------|--------|------|\n| EBA | 0.03        | 0.36  | 0.61 | 0.00   | 0.00 |\n| FFA-1  |  0.00    | 0.79  | 0.16 | 0.05   | 0.00 |\n| FFA-2 | 0.00      | 0.83  | 0.17 | 0.00   | 0.00 |\n| OPA | 0.54        | 0.12  | 0.14 | 0.05   | 0.15 |\n| PPA | 0.44        | 0.25  | 0.11 |  0.10  | 0.10 |\n| RSC | 0.66        | 0.23  | 0.11 | 0.00   | 0.00 |\n\n\n\nA unique aspect of our approach is that it provides learned queries that determine the type of information that would be routed to each ROI. We can therefore study the similarity of the query embeddings to study attention behavior (more similar query embeds lead to more similar attention maps between ROIs). Supplementary figures S5 - S8 (Sec A.2) show the cosine similarity between all of these learned ROI queries. \n\nA graded attention routing signal emerges along the hierarchy with areas becoming less spatial and more categorical. In early visual areas a checkerboard pattern emerges, where V1v in left hemisphere routes very similar information as the V2v in the same hemisphere (ventral ROIs more similar to one another than to their dorsal counterparts). Meanwhile, attention maps for higher visual ROIs are more semantically selective and form categorically selective clusters.\n\nWe believe these two analyses strengthen our claims on improved interpretability of our approach compared to alternatives. And we will highlight them more in the revision.\n\n> **3)**  Understanding the significance of an encoding accuracy\n\nAll the results presented in our paper are based on encoding accuracy normalized by noise ceiling, indicating the portion of the variance that is explained. So the ceiling is always at 1 (that's as good as we expect any model to perform). \n\n\n> **4)** Define vertex early\n\nSure, we will add the below definition and ref in section 4.1. \n\n\"In fMRI (functional magnetic resonance imaging), a vertex typically refers to a point on the surface mesh of the cortex used in surface-based analysis. It is analogous to a \"voxel\" (volumetric pixel) in volumetric fMRI data, but defined in the 2D cortical surface space.\"\n\nFischl, B. (2012). FreeSurfer. NeuroImage, 62(2), 774–781. https://doi.org/10.1016/j.neuroimage.2012.01.021\n\n> **5)** Relation to models in comp neuro models\n\nRelational processing in the visual and other perceptual systems is thought to be implemented through lateral and large-range connectivity in the primate brain. However, how brain regions flexibly access the results of each others inferential computations is not well understood. A content-specific selective access mechanism as provided by transformer attention is probably also important for the primate brain. Our work shows that including such a mechanism in a brain-computational model improves the model's ability to predict neural responses. Although the particular mechanism for flexible routing implemented in the primate brain is yet to be discovered, our work contributes to the evidence that the human brain employs some such mechanism. We will add a discussion on this in the revision and plan to use the following references to acknowledge the important links to the literature.\n\nRamalingam, N., McManus, J. N., Li, W., & Gilbert, C. D. (2013). Top-down modulation of lateral interactions in visual cortex. Journal of Neuroscience, 33(5), 1773-1789.\n\nDavid, S. V., Fritz, J. B., & Shamma, S. A. (2008). Task reward structure shapes receptive field modulation in primary auditory cortex. Nature Neuroscience, 11(6), 639–646.\n\nGilbert, C. D., & Li, W. (2013). Top-down influences on visual processing. Nature Reviews Neuroscience, 14(5), 350–363.\n\nWomelsdorf, T., & Everling, S. (2015). Long-range attention networks: Circuit motifs underlying endogenously controlled stimulus selection. Trends in Neurosciences, 38(11), 682–700.\n\nJoão D Semedo, Amin Zandvakili, Christian K Machens, Byron M Yu, and Adam Kohn. Cortical areas interact through a communication subspace. Neuron, 102(1):249–259, 2019.\n\n\n> **6)** The effect of training data size on the performance\n\nExcellent suggestion. Our models were all trained with the training split of size 8800 images. We now experiment with training the network with different fractions of the total number of training images (550, 1100, 2200, and 4400) and display the results below.\n\n**Encoding accuracy using DINOv2 backbone**\n\n| Encoder                                               | Training set size   | S1   | S2   | S5   | S7  | \n|------------------------------------------------|------|------|------|------|------|\n| Ridge regression        |8800      | 0.56 | 0.52 | 0.50 | 0.37 | \n| Transformer (rois) | 550                         | 0.41 | 0.45 | 0.45 | 0.34 | \n| Transformer (rois) |1100                       | 0.46 | 0.48 | 0.49 | 0.36 | \n| Transformer (rois) | 2200                       | 0.51 | 0.52 | 0.51 | 0.39 | \n| Transformer (rois) | 4400                       | 0.56 | 0.54 | 0.54 | 0.40 | \n| Transformer (rois) | 8800                       | 0.60 | 0.56 | 0.56 | 0.42 |\n\n\nIt is very encouraging to see that our approach can be trained using with as little as a few hundred samples making it suitable for smaller scale experiments as well. Also encouraging is to see that our model can achieve accuracy on par with the baseline models with a fraction of the training set. \n\n> **7)** Temperature parameter\n\nIn all of our experiments, we used the most common choice of the scaling value, which is $1/sqrt(d)$ and with d = 768 in our model (so ~0.036). The motivation was that this parameter in many implementations of attention layer is not easily accessible so keeping the default scaling value and optimizing other hyperparameters makes it easier to use for other researchers.\n\nHowever, to address reviewer's question, we did an experiment manipulating the scaling value from 0.001 to 1.0. For values near the default value (0.01 to 0.1) the encoding accuracy was very similar to the one from the default value, but moving to more extreme values (e.g. 1.0) led to lower performance. Although not conclusive, it appears to us that the default scaling value that has worked well across many domains seems to also be near optimal in this task. \n\n\n> **8)** MISC\n\nWe will add the missing ref, make the \"minus\" sign more clear in the figure, and will add color bar to the attention map. Thank you for pointing those out. And we hope our responses have addressed your comments, and look forward to any additional discussion!"}, {"review_id": "nHWWwbcEbp", "reviewer": "Reviewer_UAkU", "summary": "This paper introduces a new framework with the attention mechanism from a the transformer unit to model feature routing from images to various regions of the visual cortex. It shows that the transformer based framework performs better and is more interpretable than the ridge based method for understanding visual representation in the brain.", "questions": "For Figure 2b ,2c and 3b,  showing all subjects average (instead of just subject 1) with errors bar will help readers get a sense of the improvement of performances with respect to individual variability.", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n- The framework proposed in the paper is novel. \n- The framework itself is simple, yet very effective. The improvement of encoding accuracy is encouraging and this method largely fills in the gap where most encoding models ignore receptive field of the respective brain regions they are trained to model. \n- I see the method widely applicable to most studies with encoding model with minimal extra computes/training time. \n- The writing is very clear and the paper is organized well.\n\nWeakness:\n- To demonstrate the point that the attention module is \"inherently more interpretable\" I would expect the authors discover new scientific insight with this method. Attention map visualization from V2d, OFA and FBA shows the method indeed works. I am wondering whether the authors have tried visualizing less understood areas? Besides that, attention map of vertices along certain axis could be potentially very interesting maybe we could see smooth gradient of feature coding both semantically and spatially.", "was_revised": true, "final_justification": "Overall solid paper with a relatively high impact method. No unresolved issues.", "rebuttal": "We thank the reviewer for their thoughtful comments and feedback! Please see below for our responses.\n\n\n> **1)** Interpreting attention maps \n\nIn this work, we focused on the NSD-general subset of areas in the brain, comprising mostly of visually responsive vertices in the back of the brain (Fig.1 A). These vertices are mostly labeled with known categorical selectivity. However, in parallel work, we have extended our model to predict the activity for the parcels across the whole brain and we have observed the effectiveness of our method in labeling unknown regions even beyond the traditional visual cortex (e.g. ROI for tool use). \n\nHowever, to strengthen our claims on attention effectiveness, we have done experiments now to quantitatively examine the maps. We classified each pixel of the test set images into five categories using YOLOv5 (Ultralytics., 2020) and YOLOv8-face ( Dhar, A., 2023): background, face, body, animal and food. For each ROI, we calculated and resized the attention maps to $434 * 434$ for these images, and report the categories of the$~2k$ pixels with top attention values.  We found that the category selectivity is consistent with ROI labels, with EBA most selective for body, FFA most selective for face, and OPA/PPA/RSC most selective for background. \n\nBelow shows result for subject 1, we have observed similar results with subject 2, 5, 7 and will include them in the revision.\n\n|       | background | face | body | animal | food |\n|-------|------------|------|------|--------|------|\n| EBA | 0.03        | 0.36  | 0.61 | 0.00   | 0.00 |\n| FFA-1  |  0.00    | 0.79  | 0.16 | 0.05   | 0.00 |\n| FFA-2 | 0.00      | 0.83  | 0.17 | 0.00   | 0.00 |\n| OPA | 0.54        | 0.12  | 0.14 | 0.05   | 0.15 |\n| PPA | 0.44        | 0.25  | 0.11 |  0.10  | 0.10 |\n| RSC | 0.66        | 0.23  | 0.11 | 0.00   | 0.00 |\n\n\n\n> **2)** Smooth gradient of feature coding both semantically and spatially revealed by attention maps\n\nA unique aspect of our approach is that it provides learned queries that determine the type of information that would be routed to each ROI. We can therefore study the similarity of the query embeddings to study attention behavior (more similar query embeds lead to more similar attention maps between ROIs). Supplementary figures S5 - S8 (Sec A.2) show the cosine similarity between all these learned ROI queries. These similarities are also highly correlated with functional connectivity between these areas. \n\nAs the reviewer points out, the graded attention routing signal emerges along the hierarchy with areas becoming less spatial and more categorical. In early visual areas a checkerboard pattern emerges, where V1v in left hemisphere routes very similar information as the V2v in the same hemisphere (ventral ROIs more similar to one another than to their dorsal counterparts). Meanwhile, attention maps for higher visual ROIs are more semantically selective and form categorically selective clusters. We will add more discussion of this graded process in the revision. \n\n> **3)** For Figure 2b ,2c and 3b, showing all subjects average (instead of just subject 1) with errors bar will help readers get a sense of the improvement of performances with respect to individual variability.\n\nThank you for the excellent suggestion, we will include this figure in the revision. \n\nWe hope that our responses have addressed your comments and we look forward to any additional questions!"}, {"review_id": "3VsW2Yf4wq", "reviewer": "Reviewer_EjBm", "summary": "This paper presents a transformer‐based readout model for mapping DNN features to brain responses. It employs content‐dependent, attention-driven routing of retinotopic features to category-selective cortical areas. Using large-scale fMRI data (NSD), the authors show that their single-layer decoder outperforms both ridge regression and spatial-feature factorized baselines across vision and text modalities.", "questions": "The paper needs some neuroscientific grounding. It remains unclear which biological phenomenon the transformer readout captures. Citing studies on content-dependent RF shifts (e.g., attention-modulated RF dynamics) would strengthen the link to brain physiology.\n\nAlternative attention baselines: A simpler attention-modulation module (without a full transformer block) could serve as a useful comparison to isolate the benefit of cross-attention versus generic feature reweighting. For eg see Khosla et al. \n\nROI-level granularity: Applying the same dynamic routing across all voxels in an ROI may obscure within-ROI variability—particularly in early visual areas with heterogeneous RFs. This might explain the limited gains of vertex-based queries in higher early visual regions.\n\nThe spatial-feature factorized model could benefit from sparsity or smoothness constraints on learned RFs; exploring such priors might narrow the performance gap.\n\nIt is unclear how the ensemble results relate to the rest of the paper. Is the main conclusion that ensembling across layers primarily helps early visual areas? Do the authors have an intuition for why that is the case? \n\nBeyond qualitative examples, measuring whether attention peaks overlap with expected stimulus features (e.g., faces for FFA, left visual field for FBA) would substantiate interpretability claims.\n\nMinor typos:\n\n    Line 45: change “has shown that…” to “have shown that…”\n\n    Line 249: correct “TO test this” to “to test this.”\n\nRefs: Khosla, Meenakshi, et al. \"Neural encoding with visual attention.\" Advances in Neural Information Processing Systems 33 (2020): 15942-15953.", "limitations": "yes", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "- The idea is simple and elegant. Framing dynamic receptive fields as an attention routing problem is both intuitive and powerful.\n- The manuscript is well organized, with intuitive figures/results and clear explanations.\n- The method is pretty general (can be applied to transformer architectures across modalities or even CNNs) - and they demonstrate gains with both image and caption inputs to highlight the generality of the method.", "was_revised": true, "final_justification": "The authors addressed all my concerns with new baselines", "rebuttal": "We thank the reviewer for the very thoughtful review. Please see below for our responses. \n\n> **1)** Neuroscientific grounding for flexible routing\n\nThank you for this insight. A flexible routing mechanism would indeed be reflected in deviation from the classical RF characterization of responses, so content-dependent RF shifts provide some evidence for a more flexible mechanism. Another important connection is to studies of low-rank communication between brain areas which can also be modulated by attention. We will add the following references to acknowledge the important links to the literature and would appreciate any other studies you can suggest.  \n\n\nRamalingam, N., McManus, J. N., Li, W., & Gilbert, C. D. (2013). Top-down modulation of lateral interactions in visual cortex. Journal of Neuroscience, 33(5), 1773-1789.\n\nDavid, S. V., Fritz, J. B., & Shamma, S. A. (2008). Task reward structure shapes receptive field modulation in primary auditory cortex. Nature Neuroscience, 11(6), 639–646.\n\nGilbert, C. D., & Li, W. (2013). Top-down influences on visual processing. Nature Reviews Neuroscience, 14(5), 350–363.\n\nWomelsdorf, T., & Everling, S. (2015). Long-range attention networks: Circuit motifs underlying endogenously controlled stimulus selection. Trends in Neurosciences, 38(11), 682–700.\n\nJoão D Semedo, Amin Zandvakili, Christian K Machens, Byron M Yu, and Adam Kohn. Cortical areas\ninteract through a communication subspace. Neuron, 102(1):249–259, 2019.\n\n> **2)** Alternative attention baseline \n\nThanks for bringing the Khosla et al. (2020) study to our attention. It it very relevant and we agree the performance on this baseline is very informative.  \n\nTo implement this baseline, we used DeepGaze (Kummerer et al, 2022), a state the art saliency model, to generate bottom-up saliency maps for all the images in our dataset. We then resized the maps to $31*31$ and used the resulting attention values (weights) to combine the token representations to create a single token. A linear regression was then trained to map these compressed representations to vertex activation. \n\n\n**Encoding accuracy using DINOv2 backbone**\n\n| Encoder                             | S1   | S2   | S5   | S7  |\n|-------------------------------------------|------|------|------|------|\n| Saliency based integration                    | 0.38 | 0.37 | 0.44 | 0.32 |         |\n\nThe table above shows the encoding accuracy for this model across the four subjects. As you can see from our response to reviewer RcVD under **additional baselines**, the encoding accuracy of this model is comparable to the CLS+regression model. This is not surprising as the CLS token does take a weighted average of all the other tokens that are important for classification and one would expect the important tokens for classification would also have high bottom-up activation. We will discuss the saliency-based baseline in the revision as it will indeed \"serve as a useful comparison to isolate the benefit of cross-attention versus generic feature reweighting.\"\n\n\n\n> **3)** ROI-level granularity: Applying the same dynamic routing across all voxels in an ROI may obscure within-ROI variability—particularly in early visual areas with heterogeneous RFs. This might explain the limited gains of vertex-based queries in higher early visual regions.\n\nExactly matches our intuition as well. We will make this point clear in the revision. \n\n\n> **4)** Ways to improve the the spatial-feature factorized baseline model\n\nThank you for these suggestions, we did explore applying sparsity on the maps and did not see a noticeable difference in the accuracy, but we agree smoothness constraints are worth exploring as well.\n\n\n> **5)** Ensembling across layers results\n\n\nThe ensembling results show how the encoding accuracy would improve by taking into account features from earlier layers of the feature backbone. And since those features are coming from earlier layers, and we observe that the improvement are mostly in earlier visual areas, this shows further correspondence between brain areas and feature layers. It also highlights our method being suitable for ensembling as it is desired in many applications. \n\n\n> **6)** Quantitative examination of attention maps to substantiate interpretability claims.\n\nExcellent suggestion. We have done experiments now quantifying this process. We classified each pixel of the test set images into five categories using YOLOv5 (Ultralytics., 2020) and YOLOv8-face ( Dhar, A., 2023): background, face, body, animal and food. For each ROI, we calculated and resized the attention maps to $434 * 434$ for these images, and report the categories of the$~2k$ pixels with top attention values.  We found that the category selectivity is consistent with ROI labels, with EBA most selective for body, FFA most selective for face, and OPA/PPA/RSC most selective for background. \n\nBelow shows result for subject 1, we have observed similar results with subject 2, 5, 7 and will include them in the revision.\n\n\n|       | background | face | body | animal | food |\n|-------|------------|------|------|--------|------|\n| EBA | 0.03        | 0.36  | 0.61 | 0.00   | 0.00 |\n| FFA-1  |  0.00    | 0.79  | 0.16 | 0.05   | 0.00 |\n| FFA-2 | 0.00      | 0.83  | 0.17 | 0.00   | 0.00 |\n| OPA | 0.54        | 0.12  | 0.14 | 0.05   | 0.15 |\n| PPA | 0.44        | 0.25  | 0.11 |  0.10  | 0.10 |\n| RSC | 0.66        | 0.23  | 0.11 | 0.00   | 0.00 |\n\n\n\nThank you again for your great suggestions. Please let us know if you have any additional questions or comments!"}, {"review_id": "zsgFvwDPFz", "reviewer": "Reviewer_RcVD", "summary": "This work proposes a novel method for mapping activations in artificial neural networks to fRMI responses to visual stimuli. The method is based on an attention mechanism, where the model learns an input-independent query for each voxel (or ROI) which is compared to the token output of a backbone model using standard dot-product attention. The mechanism is characterized as an input-dependent receptive field that extracts only those features that are relevant to the voxel. The approach massively reduces the number of parameters of the readout module compared to other approaches and yields better predictive performance.", "questions": "1. Are both values and keys in the decoder architecture equal to the output tensor of the backbone? Fig. 1B suggests to me that key and value tensors are extracted from the backbone, but then I wouldn’t know how experiments were run on Resnet.\n2. There are quite a few typos left in the text, eg.lines 45, 48, 135, 182, 235, 237, 238, 249, 259, 266", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "**Strengths**\n\n1. The proposed method is very intuitive and easy to adapt for other researchers in the field. Therefore, I believe the paper has high relevance for the field.\n2. Predictive accuracy is higher than for the other approaches described in the manuscript.\n3. Qualitative attention maps yield results that are consistent with previously established tuning properties.\n4. The paper is well-structured and well-written.\n\n**Weaknesses**\n\n1. The algorithmic novelty is somewhat limited, as the introduction of the attention mechanism is the only novel contribution over previous work.\n2. Arguments are made that the complexity of the model does not take away from the correspondence between model activations and voxel responses (4.2). I would say that the attention mechanism is not over complex at all, but there is a lack of discussion on the fully connected layer between attention and linear map, which destroys the linearity. Is it necessary to achieve good performance?\n3. Since there is 10-fold cross validation yielding 10 models, which are averaged, it is possible that there is an ensemble effect that contributes to the good performance. As far as I understand, competing models are not ensembled.\n4. In my opinion, the paper misses two very important competing models: First, one based on the [cls] token representations which also extracts information from semantically salient regions similar to the proposed method. Second, one often reduces the dimensionality of the features in an unsupervised fashion. It would be good to compare the model to one based on PCA + Ridge.\n5. Since the algorithmic advancement is on the smaller side, it might be desirable to include multiple datasets or data modalities in the analysis.", "was_revised": true, "final_justification": "The thorough discussion of the questions I've raised has reinforced my position that this manuscript is suitable for publication. I've increased my score to 'Accept'.", "rebuttal": "We really appreciate the thoughtful review! Below we respond to all the comments and questions. \n\n> **1)** Algorithmic novelty of applying the attention mechanism\n\nAs you and other reviewers pointed out, the application of the attention mechanism is an elegant solution to the problem we set out to solve that can be easily applied in many different setting and massively cuts down on the number of parameters while improving performance by taking into account the dynamics of brain computations. We believe this contribution is significant in many ways. \n\nWe also want to highlight what other algorithmic explorations will be enabled by this approach. It connects the work to dynamic routing of information, understanding selectivity, and general connectivity of brain areas, among many other important topics in computational neuroscience and AI such as how neuronal circuits are reconfigured to dynamically route information for different purposes.\n\n\n> **2)** On the effect of linear mapping between attention and fMRI activation mapping\n\nAs you pointed out, a fully connected feedforward operation is applied to all the tokens after the attention operation. In our experiments, removing this operation did not have any noticeable impact on the performance. \n\nThe main non-linearity in our model is in the softmax applied to the scaled dot products to calculate the attention weights. We observed that removing this non-linearity had a significant impact on accuracy (~0.06 points decrease). \n\nWe will discuss these points in the revision. \n\n\n> **3)** Ensemble effect\n\nYou are right that there is some ensembling effect on the performance and for that exact reason we apply this cross-validation on all the models including all the baseline models. We agree otherwise it would not have been a fair comparison. We will make this clear in the revision. \n\n> **4)** Additional baselines ([cls] token and PCA + Ridge.)\n\n\nExcellent suggestions. We completely agree that these two baselines are important to include and thank the reviewer for pointing them out. We show their performance below in comparison to the Ridge regression and transformer attention-based encoding models. \n\n**Encoding accuracy using DINOv2 backbone**\n\n| Encoder                             | S1   | S2   | S5   | S7  |\n|-------------------------------------------|------|------|------|------|\n| Ridge regression                    | 0.56 | 0.52 | 0.50 | 0.37 |         |\n| Transformer (rois)                  | 0.60 | 0.56 | 0.56 | 0.42 |          |\n| CLS + regression         | 0.38 | 0.37 | 0.45 | 0.33 |           |\n| PCA + regression      | 0.52 | 0.47 | 0.46 | 0.34 |\n\n\nThe CLS token takes a weighted average of all the important/salient image tokens to create a compact representation of the image. But mapping this token to all the vertices in the brain does not result in good performance. The big difference to note is that our model allows each ROI or vertex to dynamically route the tokens that have relevant content for that ROI or vertex so in a way each learn to create their own \"CLS\" token based on the content of the image and the ROI/vertex selectivity. \nThe PCA based model performs better but still below the other two baselines. We will add these analyses to the revision.\n\n\n> **5)** Multiple datasets or data modalities in the analysis\n\nIn this paper we performed our experiments on the NSD dataset, as this is the largest and most commonly used dataset in the field, making comparison to other works easier. But we have explored a range of different encoding models, feature backbones, and routing mechanisms in this work. We also provided results in section 4.4 (Fig. 6) for the text modality. We believe these results strongly support our claims in this paper and we leave testing on other datasets to future work. \n\n\n> **6)** Values and keys in the decoder architecture\n\nAs shown in Fig. 1B, the value tensors are  extracted from the feature backbone. We then create positional embeddings based on the size of the feature maps (i.e. the number of tokens). The Keys are then set as values + positional embeddings. We follow the same exact process for all the models including ResNets. \n\n\nThank you again for your comments! We hope our answers have helped clarify your questions, and we look forward to any additional discussion."}]

Author rebuttal:

Meta-review: a) The authors proposes a new framework based on a transformer (attention-based architecture) to relate fMRI activity to ANN image features.

b) - The framework is novel, simple and effective and transferable to other similar applications.
- It can be used to identify attention maps in the brain
- Well written / easy to follow / intuitive figures.

c) Minor or resolve during the rebuttal period.

d) This paper is highly relevant to Neurips. All reviewers have recognized the high quality of the work both in substance and in form. The audience of this paper covers both the AI and the Neuroscience communities which at least justifies a spotlight (if not an oral).

e) The rebuttal period was helpful to resolve all the (mostly) minor concerns that the reviewers initially had about the paper improving their positive opinion towards acceptance.
Decision: Accept (spotlight)