Title: Transformer brain encoders explain human high-level visual responses
Venue: NeurIPS 2025
Accepted: yes
Keywords: Brain encoding

visual processing

transformers
Abstract: A major goal of neuroscience is to understand brain computations during visual processing in naturalistic settings. A dominant approach is to use image-computable deep neural networks trained with different task objectives as a basis for linear encoding models. However, in addition to requiring estimation of a large number of linear encoding parameters, this approach ignores the structure of the feature maps both in the brain and the models. Recently proposed alternatives  factor the linear mapping into separate sets of spatial and feature weights, thus finding static receptive fields for units, which is appropriate only for early visual areas. In this work, we employ the attention mechanism used in the transformer architecture to study how retinotopic visual features can be dynamically routed to category-selective areas in high-level visual processing. We show that this computational motif is significantly more powerful than alternative methods in predicting brain activity during natural scene viewing, across different feature basis models and modalities. We also show that this approach is inherently more interpretable as the attention-routing signals for different high-level categorical areas can be easily visualized for any input image. Given its high performance at predicting brain responses to novel images, the model deserves consideration as a candidate mechanistic model of how visual information from retinotopic maps is routed in the human brain based on the relevance of the input content to different category-selective regions. Our code is available at \href{https://github.com/Hosseinadeli/transformer_brain_encoder/}{https://github.com/Hosseinadeli/transformer\_brain\_encoder/}.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a novel method for mapping activations in artificial neural networks to fRMI responses to visual stimuli. The method is based on an attention mechanism, where the model learns an input-independent query for each voxel (or ROI) which is compared to the token output of a backbone model using standard dot-product attention. The mechanism is characterized as an input-dependent receptive field that extracts only those features that are relevant to the voxel. The approach massively reduces the number of parameters of the readout module compared to other approaches and yields better predictive performance.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The proposed method is very intuitive and easy to adapt for other researchers in the field. Therefore, I believe the paper has high relevance for the field.\n2. Predictive accuracy is higher than for the other approaches described in the manuscript.\n3. Qualitative attention maps yield results that are consistent with previously established tuning properties.\n4. The paper is well-structured and well-written.\n\n**Weaknesses**\n\n1. The algorithmic novelty is somewhat limited, as the introduction of the attention mechanism is the only novel contribution over previous work.\n2. Arguments are made that the complexity of the model does not take away from the correspondence between model activations and voxel responses (4.2). I would say that the attention mechanism is not over complex at all, but there is a lack of discussion on the fully connected layer between attention and linear map, which destroys the linearity. Is it necessary to achieve good performance?\n3. Since there is 10-fold cross validation yielding 10 models, which are averaged, it is possible that there is an ensemble effect that contributes to the good performance. As far as I understand, competing models are not ensembled.\n4. In my opinion, the paper misses two very important competing models: First, one based on the [cls] token representations which also extracts information from semantically salient regions similar to the proposed method. Second, one often reduces the dimensionality of the features in an unsupervised fashion. It would be good to compare the model to one based on PCA + Ridge.\n5. Since the algorithmic advancement is on the smaller side, it might be desirable to include multiple datasets or data modalities in the analysis.\n\nQUESTIONS:\n1. Are both values and keys in the decoder architecture equal to the output tensor of the backbone? Fig. 1B suggests to me that key and value tensors are extracted from the backbone, but then I wouldn’t know how experiments were run on Resnet.\n2. There are quite a few typos left in the text, eg.lines 45, 48, 135, 182, 235, 237, 238, 249, 259, 266\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents a transformer‐based readout model for mapping DNN features to brain responses. It employs content‐dependent, attention-driven routing of retinotopic features to category-selective cortical areas. Using large-scale fMRI data (NSD), the authors show that their single-layer decoder outperforms both ridge regression and spatial-feature factorized baselines across vision and text modalities.\n\nSTRENGTHS AND WEAKNESSES:\n- The idea is simple and elegant. Framing dynamic receptive fields as an attention routing problem is both intuitive and powerful.\n- The manuscript is well organized, with intuitive figures/results and clear explanations.\n- The method is pretty general (can be applied to transformer architectures across modalities or even CNNs) - and they demonstrate gains with both image and caption inputs to highlight the generality of the method.\n\nQUESTIONS:\nThe paper needs some neuroscientific grounding. It remains unclear which biological phenomenon the transformer readout captures. Citing studies on content-dependent RF shifts (e.g., attention-modulated RF dynamics) would strengthen the link to brain physiology.\n\nAlternative attention baselines: A simpler attention-modulation module (without a full transformer block) could serve as a useful comparison to isolate the benefit of cross-attention versus generic feature reweighting. For eg see Khosla et al. \n\nROI-level granularity: Applying the same dynamic routing across all voxels in an ROI may obscure within-ROI variability—particularly in early visual areas with heterogeneous RFs. This might explain the limited gains of vertex-based queries in higher early visual regions.\n\nThe spatial-feature factorized model could benefit from sparsity or smoothness constraints on learned RFs; exploring such priors might narrow the performance gap.\n\nIt is unclear how the ensemble results relate to the rest of the paper. Is the main conclusion that ensembling across layers primarily helps early visual areas? Do the authors have an intuition for why that is the case? \n\nBeyond qualitative examples, measuring whether attention peaks overlap with expected stimulus features (e.g., faces for FFA, left visual field for FBA) would substantiate interpretability claims.\n\nMinor typos:\n\n    Line 45: change “has shown that…” to “have shown that…”\n\n    Line 249: correct “TO test this” to “to test this.”\n\nRefs: Khosla, Meenakshi, et al. \"Neural encoding with visual attention.\" Advances in Neural Information Processing Systems 33 (2020): 15942-15953.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper introduces a new framework with the attention mechanism from a the transformer unit to model feature routing from images to various regions of the visual cortex. It shows that the transformer based framework performs better and is more interpretable than the ridge based method for understanding visual representation in the brain.

STRENGTHS AND WEAKNESSES:
Strengths:
- The framework proposed in the paper is novel. 
- The framework itself is simple, yet very effective. The improvement of encoding accuracy is encouraging and this method largely fills in the gap where most encoding models ignore receptive field of the respective brain regions they are trained to model. 
- I see the method widely applicable to most studies with encoding model with minimal extra computes/training time. 
- The writing is very clear and the paper is organized well.

Weakness:
- To demonstrate the point that the attention module is "inherently more interpretable" I would expect the authors discover new scientific insight with this method. Attention map visualization from V2d, OFA and FBA shows the method indeed works. I am wondering whether the authors have tried visualizing less understood areas? Besides that, attention map of vertices along certain axis could be potentially very interesting maybe we could see smooth gradient of feature coding both semantically and spatially.

QUESTIONS:
For Figure 2b ,2c and 3b,  showing all subjects average (instead of just subject 1) with errors bar will help readers get a sense of the improvement of performances with respect to individual variability.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 4
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-07-03

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes and investigates the use of transformer encoder-decoder models in predicting fMRI responses of subjects seeing images from the Natural Scene Dataset. These transformer models outperform linear mapping/ridge-regression based on transformer-features for several different \"feature backbones.\" The authors also find higher performance for vertex-based routing vs. ROI-based routing, interpretable attention maps, and some generalization to text (BERT backbone).\n\nSTRENGTHS AND WEAKNESSES:\nI'm happy to recommend this work for acceptance. The authors are working on a highly relevant problem in neuroscience and propose a simple yet powerful alternative to the \"standard\" linear encoding approaches to predicting brain activity. The main claims of the paper are supported by experiments from multiple encoding \"backbones\". The explanations and motivations for different design choices is also well explained in the paper.\n\nI think the work could benefit further by explaining how to translate higher predictive performance into qualitative insights for particular ROIs or voxels. I recognize the supplement (Figure S9) is making an attempt at this, but I think it would also be helpful to give insights into why the transformer is performing better than linear regression in certain cases. Another idea is to go further into detail into the improved interpretability relative to other approaches.\n\nQUESTIONS:\n1) Is there a simple way to understand the significance of an encoding accuracy between for example 0.6 and 0.8, as seen in Figure 2C? For example, one way to get at this qualitatively could be by evaluating quality of decoding based on the given encoding model. It would also be helpful to state what the best explained variance one might reasonably expect on this data (e.g., due to noise).\n\n2) Can the authors provide a definition of vertex (ideally upon first use in Section 4.1)?\n\n3) The authors write, \"Our approach proposes a mechanistic model of how visual information from retinotopic maps can be routed based on the relevance of the input content to different category-selective regions.\" Could they comment on how this model could be related to experiments in systems neuroscience, or models in computational neuroscience such as dynamic routing?\n\n3) How does the performance of transformer-based methods scale with the amount of data used?\n\n4) How much of an impact do parameters like temperature impact encoding accuracy?\n\nMinor comments: \n1) possible missing ref on Line 134 \n2) For Figures 3b/4c, though the caption makes clearer that the dash should be read as \"minus\", it would be nice to make this clearer from the subtitle itself.\n3) Is it possible to add some context/colorbar for the attention maps in Figure 5?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}