Meta-Learning an In-Context Transformer Model of Human Higher Visual Cortex
Authors: Muquan Yu
Mu Nan
Hossein Adeli
Jacob S. Prince
John A. Pyles
Leila Wehbe
Margaret Marie Henderson
Michael J. Tarr
Andrew Luo
Venue: NeurIPS 2025 poster ()
Keywords: fMRI
brain
visual cortex
neuroscience
visual neuroscience
NeuroAI
visual selectivity
semantic selectivity
in-context learning
Primary area: neuroscience_and_cognitive_science
TL;DR: We meta-learn a transformer based in-context learning fMRI visual cortex encoder which can adapt to new human subjects without any fine-tuning

Abstract: Understanding functional representations within higher visual cortex is a fundamental question in computational neuroscience. While artificial neural networks pretrained on large-scale datasets exhibit striking representational alignment with human neural responses, learning image-computable models of visual cortex relies on individual-level, large-scale fMRI datasets. The necessity for expensive, time-intensive, and often impractical data acquisition limits the generalizability of encoders to new subjects and stimuli. **BraInCoRL** uses in-context learning to predict voxelwise neural responses from few-shot examples *without any additional finetuning* for novel subjects and stimuli. We leverage a transformer architecture that can flexibly condition on a variable number of in-context image stimuli, learning an inductive bias over multiple subjects. During training, we explicitly optimize the model for in-context learning. By jointly conditioning on image features and voxel activations, our model learns to directly generate better performing voxelwise models of higher visual cortex. We demonstrate that BraInCoRL consistently outperforms existing voxelwise encoder designs in a low-data regime when evaluated on entirely novel images, while also exhibiting strong test-time scaling behavior. The model also generalizes to an entirely new visual fMRI dataset, which uses different subjects and fMRI data acquisition parameters. Further, BraInCoRL facilitates better interpretability of neural signals in higher visual cortex by attending to semantically relevant stimuli. Finally, we show that our framework enables interpretable mappings from natural language queries to voxel selectivity.

Meta-Learning an In-Context Transformer Model of Human Higher Visual Cortex
Muquan Yu
Mu Nan'
Hossein Adeli
Jacob S. Prince
John A. Pyles
Leila 6ehbe
Margaret M. Henderson
Michael J. Tarr
Andrew Luo" #1
University of Hong Kong Chinese University of Hong Kong
Columbia University
Harvard University
UUniversity of Washington
Carnegie Mellon University
mqyuělink. cuhk. edu. oahk.edd..ba hk
'Corresponding author: aluo@hku.hi .k
Abstract
Understanding functional representations within higher visual cortex is fun- damental question in computational neuroscience. While artificial neural net- works pretrained on large-scale datasets exhibit striking representational align- ment with human neural responses, learning image-computable models of vi- sual cortex relies on individual-level, large-scale fMRI datasets. The necessity for expensive, time-intensive, and often impractical data acquisition limits the generalizability of encoders to new subjects and stimuli. BraInCoRL uses in- context learning to predict voxelwise neural responses from few-shot examples without any additional finetuning for novel subjects and stimuli. We leverage a transtormer architecture that can flexibly condition on variable number of in- context image stimuli, learning an inductive bias over multiple subjects. During training, we explicitly optimize the model for in-context learning. By jointly con- ditioning on image features and voxel activations, our model learns directly generate better performing voxelwise models of higher visual cortex. We demon- strate that BraInCoRL consistently outperforms existing voxelwise encoder designs in a low-dat low-data regime when evaluated entirely novel images, while also exhibit- ing strong test-time scaling behavior. The model also generalizes to an entirely new visual fMRI dataset. which uses different subjects and fMRI data acquisition parameters. Further, BraInCoRL facilitates better interpretability of neural signals in higher visual cortex by attending semantically relevant stimuli. Finally, show that our framework enables interpretable mappings from natural language queries voxel selectivity. Our code and model weights are publicly available https:/ // / thub. com cow/leomqyu/ Baragyuu/raaCC.......
Introduction
Human visual cortex transforms raw sensory input into behaviorally-relevan representations of the world. While early visual areas are characterized by retinotopic organization and selective tuning to simple features such as edges and orientation gradients [1, 2, 3, 4, 5], higher-order visual areas demonstrate selectivity more abstract semantics and categories. While this functional organization is largely consistent across individuals at a coarse scale, the spatial distribution and fine- grained semantic selectivity within visual cortex varies due to structural differences. developmental experience, and life-long learning [6, 7, 8, 9, 10, 11]. Such functional inter-subject differences pose a fundamental challenge in constructing generalizable models of higher visual cortex that can adapt subject-specific neural organization without exhaustive data collection for every individual.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Recent advances in deep learning offer promising avenue for addressing this challenge. Vision models pretrained on large-scale image datasets not only achieve strong object recognition perfor- mance, but also recapitulate hierarchical processing patterns observed in biological vision [12, 13, 14]. While these models may encapsulate some universal principles of visual processing [15], they do not inherently account for individual differences in cortical organization. To close the gap between artificial and biological systems, researchers have developed image-computable IMRI encoders - models that predict brain activity from visual stimuli [16]. These encoders typically regress image features onto voxelwise brain responses using subject-specific data, acting as computational probes of visual processing. Unfortunately, current approaches require many hours of costly fMRI scans per subject to fit these mappings a prohibitive bottleneck for scalability to new populations, stimuli, and tasks, especially in clinical settings where collecting large amounts of data is difficult.
We bridge this gap with BraInCoRI. (Brain In- Context Representation Learning). a transformer- based framework that meta-learns to predict subject-specific neural responses from provided examples. Inspired by language models that adapt to new tasks in-context, our approach treats voxel encoding as function inference problem: given handful of stimulus-responss pairs from new individual and novel stimuli, BraInCoRL constructs a voxelwise encoding model without any further training. By jointly optimizing for in-context learning across diverse subjects and stimuli, our model discovers shared functional principles of higher visual cortex that generalize to new subjects and stimuli represented by only small amount of data. We illustrate the problem we are tackling in Figure 1.
We demonstrate that BraInCoRL: (1) Outperforms existing voxelwise encoder models in the low-data regime on novel visual stimuli while exhibiting strong generalization with increasing context. (2) Can generalize to new experiments with different scanning parameters. (3) Through analysis of attention values, learns to rely on images that are reflective of the category selected for in each region. (4) When paired with features from contrastive image-language models, facilitates zero-shot natural language-based characterization of cortical selectivity, enabling interpretable, finer-grained query-driven functional mapping.
2 Related work
Computational Encoding and Decoding Models for Visual Cortex, Computational modeling of neural data often involves two complementary approaches: encoding models that map from stimuli to neural activations, and decoding models that map from neural data to stimuli [16, 17, 18, 19, 20, 21, 22, 23, 24]. The development of both approaches has been facilitated by advances in machine learning models. For encoding models, the dominant approach is to combine pretrained deep feature extractors with linear voxelwise weights [25, 26, 27, 28, 29, 30J. More recent approaches have proposed to leverage transformers [31, 32] to learn the relationship between brain regions of a single subject. Most similar to our framework is the pioneering work by Adeli al. [31] and Beliy& & Wasserman et al. [33] which uses an auto-decoder based transformer network for multi-subject voxelwise encoding; However these approaches still require fine-tuning for novel subjects. More generally, encoders have been used to investigate the coding properties in higher-order visual areas [34, 35, 36, 37, 38, 39, 40, 41]. Encoders have been further combined with image- generation models [42, 43, 44, 45, 46, 47, 48, 49] or Eanguage-generatin
models [50, 51] to explore semantic selectivity. Recent progress on large generative models has enabled stimulus reconstruction from fMRI, EEG, and MEG signals for images [52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63], videos [64, 65, 66, 67, 68, 69, 70], and language/audio [71, 72, 73, 74, 75, 76, 77].
Representational Organization of Visual Cortex. Human visual cortex exhibits a hierarchical organization from primary visual to higher-order visual areas. The higher visual cortex is characterized by tiling of semantically specialization. Approaches using functional localizers have identified category selective regions in higher visual that are responsive to faces [78, 79, 80, 81], places [82, 83), bodies [84], objects [85, 86], food [87, 88, 89], and words [90, 91]. While the spatial location regions are generally consistent across subjects [92], significant
of these broad category-selecee inter-individual variability exists in their anatomical location, spatial extent, and response profiles [6,
7, 8, 9, 93, 94, 95, 96, 97]. Accurately characterizing visual processing in higher-order visual areas necessitates subject-specifit encoding models that capture individual diversity [98].
Meta-Learrning and In-Context Learning. Our work builds upon meta-learning and in-context learning (ICL). Meta-learning trains models to "leam to learn" from a distribution of tasks, enabling

---

(a)
Visual Stimuli
Activations
Unobserved
True Response
Function
Visual Dimension
Noisy Measurement
Novel Stimuli
Visual Dimension
Other dims
(b)
Visual stimuli seen by subject
Space of response functions (differs by voxel & subject)
Voxel-wise function
Responses
New response O000 Model Input
estimation
Subj
Input
Measured
BrainCoRL
Subj
New stimuli
Subj
Meta-t trainiriinngggg tasks
Training-free in-context adaptation to novel subject
(c)
(d)
Expl. Variance
5-2x
1.3x
8.12
Ridge Baseline
Fully Tra ned
Fully Trained
BraInCoRL
Ridge Regression
In-Context Samples
Meta-Learning In-Context Visual Cortex Encoder. (a) The voxelwise brain encoding problem. For each voxel, there a response function that maps from visual stimuli to voxel activation. In practice, is to infer an image-computable function for each voxel to predict its activation. (b) BraInCoRL treats each voxel meta-learning task, and samples (image, response) pairs from multiple subjects. During testing, the model is conditioned on a small number of novel images and measurements from subject and directly outputs the function parameters. (c) From left right, the explained variance from the full model trained on 9,000 images from one subject, BraInCoRL. with only 100 in-context images from the new subject, and baseline ridge regression also with 100 images (for this baseline, voxelwise regularization is determined using 5-way cross-validation). Our method achieves much higher data efficiency than baseline. (d) Explained variance as a function of in-context support set size. As the in-context support set size increases from 0 to 1,000, BraInCoRL steadily improves and approaches the fully trained reference model fit converge on each subject's full 9,000-image training set, demonstrating high prediction accuracy and data efficiency.
Figure 1: BraInCoRL:
only observe the noisy measurements from fMRL. The goal
quick adaptation new tasks with few examples, via methods like meta-optimization [99, 100, 101] or metric-based approaches [102]. More recently, ICL has emerged as a powerful capability in Large Language Models [103, 104], where models adapt to new tasks inference time solely based on examples provided in their prompt without any parameter updates [105, 106]. This has led to hypotheses that ICL an emergent form of implicit meta-learning, where transformers effectively learn underlying learning algorithms during pre- pre-training [107, 108). Our goal is to learn the structure of functions that map between visual stimuli and voxelwise brain response. Our framework combines meta-training (across voxels and subjects) and in-context learning (across stimuli) to enable training free adaptation to novel subjects.
Methods
Our proposed framework leverages meta-learning and uses few-shot, in-context examples for vox- elwise prediction of unseen stimuli (Figure 2). Critically, for unseen subjects, this approach does not require any additional finetuning. We achieve this by treating the mapping function from visual stimuli and the response of individual voxels as a set of meta-training tasks. This voxelwise approach is in line with higher-order visual areas being described by multitude of functionally diverse voxels, which we randomly sample during training.
3.1 Motivation and Problem Definition
There is substantial inter-subject anatomical and functional variability in the higher visual cortex among humans. Consequently, while one can learn per-subject image-computable encoders that map image features to brain responses with high predictive accuracy, these models require large amounts

---

In-Context Conditioned Higher Visual Cortex Transformer
Voxelwise Encoder
Generated
Encoder Weights
Per-Image Non-Linear Feature Extractor
Image
Embedding
Voxel
Activation
In-Context Stimuli
Novel Stimuli
Figure 2: Architecture of the In-Context Voxelwise Encoder (BraInCoRL). (1) A pretrained feature extractor converts visual stimuli into vector embeddings. (2) A higher visual cortex transformer integrates these embeddings with voxel activations leam context-specifie features and generates hyperweights for subsequent voxelwise encoder backbone. (3) The voxelwise encoder, conditioned on the hyperweights, predicts voxel responses for novel stimuli.
of within-subject data and do not exploit information across subjects. To account for this variability across individuals, design framework treat individual voxels the fundamental unit of modeling. Importantly, our method does not assume any overlap in stimuli across subjects, yet still enables to take advantage of multi-subject training data.
We formalize this problem by assuming an image j is represented as RGB tensor Ij E xx Given an image j and a human subject k, there is 1D array of voxel activations (beta values) from higher visual cortex: (B1, between subjects.
Br,)j = Br & E II where the number of voxels will differ
Given new subject not during training, have small set of images (I1, I2, .... In) and measured brain responses (B1, B..... B,. for this new subject. Our goal is to estimate the brain response to an arbitrary new image Inovel-
3.2 Meta-Learning an In-Context Transformer
camue-computable encoders that map from images to brain responses for a single subject are typically considered function fie (I) B, and jointly model the entire visual cortex. While powerful, this approach cannot be easily extended to the multi-subject scenario, where test-time individuals may have functional and anatomical differences that not known during training. In contrast, BraInCoRI. considers each voxel v to have a unique and unobserved visual response function (Z) > 3r. Voxels can be from many different subjects. During training. we consider each voxel's response function to be meta-training task, where each task is effectively specified by input images and voxel response pairs. In order to facilitate training-free ffee adaptation on new subjects, utilize in-context learning across stimuli enabled by a transformer backbone.
For single voxel we define {(21, (11), (x2, ,2,, (Ip, Bp)]. where 2; E RT is the image embedding vector extracted by frozen image feature extractor o(-), i... 2; o(I,), and B R is the voxel's response observed for image I,. Each pair is concatenated to form context tokens C; = [2 8.]. and the full context is defined {cI.-... Cp). Unlike traditional in-context inference in large language models, where there is query concatenated to the end of the context, we avoid expensive per-voxel inference by directly generating the parameters for the voxelwise visual response function. During training, we optimize the BraInCoRL transformer 7' with parameters 0 such that it outputs voxel response function f with parameters
support set of P images and neural responses
w = To 02. .p,
8 = (T)
(2)
Since T and f are differentiable, optimize 0 maximize the likelihood of observing 8 given I:
0" arg min ll
S
In practice, mean-squared-erro and gradient based mini-bate batch optimization.

---

Table 1: Voxelwise performance across five category-selective regions. Explained variance is shown for our in-context model ("BraInCoRL.") that uses just 100 in-context images, the fully trained reference model fit to converge on each subject's s full 9,000-image training set ("Fully Trained"), and within- subject ridge regression baselines (100, 300 within- subject test images), plus the FsAverage map averages over other subjects, Our model outperforms both subject-wise and anatomical baselines, and demonstrates strong data-efficiency.
Faces
Places
Bodies
Words
Food
Mean
S2
S1
S2
S2
S1
S2
S1
S2
Fully Trained
0.19
0.16
0.20
0.27
0.28
0.24
0.11
0.11
0.16
0.17
0.18
0.19
Ridge-100
0.10
0.07
0.08
0.14
0.16
0.12
0.02
0.03
0.05
0.07
0.07
0.08
Ridge-300
0.13
0.10
0.13
0.20
0.22
0.16
0.06
0.06
0.10
0.11
0.11
0.12
FsAverage map
0.13
0.06
0.11
0.19
0.09
0.08
0.06
0.03
0.14
0.18
0.08
0.06
BraInCoRI--100
0.16
0.13
0.16
0.23
0.25
0.21
0.07
0.08
0.12
0.13
0.13
0.15
3.3 Test-time Context Scaling
At test time, when we encounter new subject, we assume have access small set of novel images and the corresponding brain responses we want to predict a voxelwise encoder. While our voxelwise parameterization successfully resolves the challenge of unknown test-time anatomy and geometry, the challenge of unknown test-time context size remains, Unlike transformers in language models, where the output dependent on the order of the samples, we want our model to be explicitly invariant to input order. In order to facilitate variable length context, we utilize logit scaling [109, 110, 111]. Assuming query/key (g, k) with de features and a length context:
9- Vike
log (1) q k ddg
Okorig
Osscaled
We find this method effectively enables context scaling when trained with variable length context. While the hypernetwork could, in principle, parameterize any class of neural encoders (e.g., MLPs, convolution, attention layers), prior studies utilizing brain data have largely used linear parameteriza- tions that map from deep network features to voxel responses [13, 14], and find that such a choice offers high performance and interpretability. Given features z E R from a pretrained neural network ((Z), we adopt the same setup and predict the final voxel response:
B w)) matmul (x:,w)
(5)
more detailed description on the test-time context scaling technique provided Appendix A.4.
4 Experiments
We utilize BraInCoRL to generate encoder weights in low-data regime. We start by describing our experiment setup. First, we evaluate the effectiveness of BraInCoRL on novel subjects where there is zero overlap between the training dataset and the evaluated subject's in-context stimulus. We also evaluate our framework where data from novel subjects are collected from a completely different scanner and protocol. Second, we explore the attention pattern across stimuli for different ROIs, and perform ablations to explore the need for test-time in-context stimulus diversity. Third, we show that our method enables natural language characterizations of the visual cortex using very little data.
4.1 Setup
Dataset. We primarily perform experiments with the Natural Scenes Dataset (NSD) [112], but then validate with the BOLD5000 dataset [113]. Both are large-scale scale neural datasets: NSD is the largest 7T fMRI image viewing dataset available, where eight subjects each viewed ~ 10, 000 images; BOLD5000 is 3T dataset, where four subjects each viewed ~ 5000 images. In NSD each image was viewed up to three times, while in BOLD only subset of images were viewed up to four times. For NSD, of the eight subjects, four subjects (S1, $2, S5, S7) completed scanning and are the focus of our analysis in the main paper. The results of other subjects presented in the supplemental
5

---

(a)
Subj2
Subj5
Subj7
Expl. Variance
10k
1.15
10x
0:10
BrainCoRL
0
Ridge Baseline
Fully Trained
250
500
750
1000
0
250
500
750
1000
250
500
750
1000
(b)
# In-Context Samples
# In-Context Samples
# In-Context Samples
Expl. Variance
0.25
Ridge
Pretrained
0.20
BrainCoRL HO
BrainCoRL no HO
0.15
0.10
0.00
Subj1
Subj2
Subj5
Subj7
(c)
Fully Trained
y=x
y=x
0.40
0.20
0.00
CLIP
DINO
SigLIP
Figure 3: Evaluation on NSD. (a) Prediction explained variance of BraInCoRL improves on novel subjects with larger in-context support set size, outperforming within vithin-subject ridge regression and approaching the fully trained reference model fit on each subject's 's full 9,000-image training set, using far less data. (b) Ablation (100 support images) comparing BraInCoRL variants: the original model trained while holding out the novel subject's 's 9,000 test-time support images ("HO"), a BraInCoRL model trained without this holdout ("no HO"), and pretraining-onl BraInCoRL model, alongside the within- subject ridge baseline. Results show that finetuning with real fMRI data improves performance, and holding out the test subject's image data does not hinder generalization. (c) Voxelwise explained variance from BraInCoRL (100 images) is strongly correlated with fully trained reference models across different visual encoder backbones. Note that the y-axis represents explained variance of the fully trained model (9,000 images), while x-axis represents explained variance of BraInCoRL.
materials. For each subject, ~ 9, 000 images are unique to each other, while ~ 1, 000 are viewed by all eight subjects. The NSD stimuli were sourced from the MS COCO dataset (as were a subset of the BOLD5000 stimuli). Unless otherwise noted, we perform our analysis in subject native volume space (funcipt8mm) for NSD, where the voxelwise betas are z-scored within each session then averaged across repeats of the same stimulus. In order to rigorously evaluate BraInCoRL for a given subject, the xx9 9,000 unique images viewed by the other three subjects the meta-tran trainnta data. During the ROI-wise evaluation for NSD, we follow prior work [48] and apply a f-statistic cutoff of t >> by leveraging independent functional localizer data provided with the dataset threshold the originally broad definitions. During quantitative evaluation, we follow prior work [1 [114] and apply a voxel quality cutoff of ncsnr 0.2. For BOLD5000,
model trained four NSD subjects (S1, S2, S5, S7). Following the suggestion of BOLD5000 authors, we only model stimuli with repeats and apply cutoff of nesnr 0.3. Voxel responses are averaged across repeats.
Training and evaluation. Our training process takes inspiration from LLM based training setups, and we adopt three stage process pretraining, context extension, and supervised fine tuning. In the pretraining stage, we use an analysis-by-see by-stthesiss approach without relying on any (real) subject data. We artificially construct a large number of voxels with synthesized weights. We derive synthetic voxel responses with normally distributed noise using these synthesized weights and train our model using fixed context size of 500. In the second stage, we randomly sample the context size from Uniform(3l (300, 500) which allows the model to acquire length robustness. Finally, in the finetuning stage, the model is trained real fMRI data using the subject-specific beta values,

---

UIMAP1
OP
FBA
Food
FEA
FEA
region
Anterior
Anterior
2TL
Left Hemisphere
faues
Right Hemisphere
(b)
PPA
(c)
$2
S5
S7
Figure 4: UMAP visualization of predicted response weights. We apply UMAP to BraInCoRL -predicted voxelwise weights (100 support images) and show: (a) a flatmap for S1 with ROI outlines, (b) the projection clusters align with body/face regions (EBA, FFA/aTL-faces) place regions (RSC, OPA, PPA), and food regions (in red).
inflated surface, and (c) flatmaps for S2, S5, and S7. Color-coded
enabling adaptation to biologically grounded neural responses. All our evaluation experiments are performed on novel subjects that are unseen by the model during training, with exception of (Fig- ure 3b) no heldout ("no HO"") ablation study.
4.2 Effectiveness of In-Context Higher Visual Cortex Encoders
On NSD. In this experiment, evaluated BraInCoRL using each subject's 9,000 unique images as the in-context support set and the shared 1,000 images as the test set. For each evaluation, we randomly sampled training images from the subject-speciftt in-context support set and test images from the shared test set. Explained variance statistics averaged over dategory-sslectivv ROIs are reported in Table 1. We compare BraInCoRL using just 100 images against within-subjes ridge regression baseline trained on 100 and 300 support-set images of the test-subject, with the regularization strength selected via 5-fold cross-validation over [10-3 10-2 ,.... 108]. Remarkably, BraInCoRI. with only 100 images nearly matches the performance of the fully supervised reference model that is trained by gradient descent on each subject's 's entire subject support-set of 9,000 images until convergence. We also evaluate anatomical FsAverage bascline which aligns each training subject's anatomy to a common template and projects the average response onto novel subjects for prediction. While this baseline benefits from strong anatomical prior, it is outperformed by BraInCoRL, which directly adapts to each subject's unique neural responses with higher efficiency.
To evaluate test-time behavior, we assess how performance scales with increasing in-context support size. BraInCoRI. consistently outperforms within-subject ridge regression and more efficiently approaches the fully trained reference model (Figure Ic for subject and Figure 3a for subject 2, 5, 7). Moreover, we conduct ablations by evaluating BraInCoRI. model trained without holding out the test subject's support images and BraInCoRL model with only pretraining. Results confirm

---

CSI1
CSI2
CSI3
0.20
2.7x
1.3x
1.3x
3.9k
Pearson
0:10
0.05
BrainCoRL
Ridge Baseline
0.00
60
80
20
20 40 60 80
In-Context Samples
In-Context Samples
In-Context Samples
Figure 5: Evaluation on BOLDS000. We evaluate BraInCoRL on the BOLD5000 dataset, which was collected using a different scanner than NSD. For varying in-context support set sizes, we report voxelwise Pearson correlation between predicted and true responses for both BraInCoRL and wi Place Voxels
Word Voxels
STOP
STOP
Face Voxels
Body Voxels
Food Voxels
Figure 6: Top contributing support images for each category-selective region in S1. For each of the five category-selectin regions, we select the in-context support images with the highest attention weights in BraInCoRL's 's final attention layer for voxels in that region. We visualize the top 5 contributing images for the place, word, face and body regions, and the top 10 for the food region.
that finetuning with real neural data boosts performance and that BraInCoRL can generalize well to previously unseen images without overfitting (Figure 3b). Additionally, we observe high voxelwise explained variance correlation between BraInCoRL and the fully trained reference model across multiple backbones (Figure 3c). Finally, we apply UMAP to the BraInCoRL predicted response- function weights, revealing clear semantic clustering across higher visual areas (Figure 4) that correspond with known visual regions.
On BOLD5000. We validate generalization on the BOLD5000 dataset in Figure 5. BOLD5000 has many differences with NSD and represents the challenge of cross-site generalization that is the main objective of our method. BOLD5000 was collected on a 3T scanner with a different stimulus presentation time, slow-event related trial structure (10s inter-trial interval), different images and image datasets, different voxel size (2mm isotropic), and different subjects. BraInCoRL achieves higher voxelwise Pearson correlations than within-subject ridge regression. Moreover, results remain consistent across different subjects, demonstrating the robustness and reliability of our method.
4.3 Semantic Discovery through Text-Image Embedding Alignment
To better understand how BraInCoRL. leverages in-context examples, we analyze internal attention mechanisms to identify images that strongly influence voxel predictions in category-selective regions. In Figure 6, we examine attention weights from BraInCoRL's final attention layer to determine the top-contributing images for each cortical region. The visualized images with the highest attention scores closely align with known semantic preferences of the respective cortical regions.
However, Figure reveals counterintuitive finding regarding the semantic specificity of in-context support sets. We systematically vary the specificity of the 100-image sets provided the model, ranging from highly relevant to random selections. Selections are determined via the first text rext-prompt each category (see Appendix).
observe that randomly selected images lead to better predictive performance compared to sets composed solely of highly relevant images. This suggests that overly specific context sets may limit the generalization capabilities of the encoder system, and diverse, less semantically constrained images provide richer context for learning robust voxel representations.

---

Food
Words
0.20
Random
Best 100
Best 201 500
0.15
Best 102-200
301-400
ExpL.
0.05
0.00
55
Places
Faces
0.30
0.20
0.10
0.00
S1
S2 55 S7
S1
S2 55
S7
Figure 7: Impact of support-set specificity on category-selective ROI encoding performance on NSD. We construct in-context support of 100 images based on descending semantic relevance for each ROI (tiers: 1-100, 101-200, 201-300, 301-400) and compare them with randomly sampled sets of equal size. Mean explained variance in the target dategory-selectioe specificity decreases, with all curated sets performing worse than random sampling. This suggests that overly specific support sets hinder generalization in voxelwise encoding. This pattern echoes prior findings on diverse stimuli contributing to better encoders [13].
ROIs increases semantic
vmin
VIMAX
probing
"A photo torso
"A photo person's
"A photo of bedroom
photo words"
Figure 8: Predicting cortical responses from natural language prompts. For each semantic category, convert natural language prompt into CLIP text embedding, project it into the image feature space, and use BraInCoRL. to predict the corresponding voxel activation map. The pre- dicted activations align closely with t-statistic of category- regions (derived from fMRI functional localizer experiments), illustrating the potential for efficient, language-driven functional mapping of visual cortex.
Table Voxelwise prompt classification accuracy. Each cell shows the percentage of voxels in given category selective region (columns) whose peak predicted activation was elicited by a specific semantic prompt (rows, see Appendix). Using only 100 support images, BraInCoRL effectively localizes category-selective regions with high data efficiency.
Bodies
Faces
Places
Food
Words
S1
S1
$2
S2
S1
$2
Bodies
0.63
0.54
0.30
0.16
0.05
0.03
0.15
0.19
0.43
0.17
Faces
0.30
0.25
0.60
0.56
0.05
0.01
0.07
0.04
0.15
0.16
Places
0.02
0.09
0.02
0.05
0.81
0.88
0.10
0.07
0.05
0.10
Food
0.04
0.10
0.08
0.18
0.08
0.06
0.66
0.64
0.31
0.45
Words
0.01
0.03
0.00
0.04
0.01
0.02
0.02
0.05
0.05
0.12

---

4.4 Characterizing higher visual cortex with text embeddings to images
In this experiment, we investigate the capability of BraInCoRL to enable interpretable, query-driven functional mapping using natural language prompts.
In Figure 8, demonstrate that natural language prompts can be effectively mapped to predicted voxel activations. For each category selective region, we convert the corresponding natural language prompt into CLIP embedding and project it into the image feature space directly predict voxel activations. The resulting activation maps closely match expected t-statistics, reflecting BraInCoRL'ss ability support intuitive, language-driven cortical queries.
In the second analysis (Table 2), we quantitatively assess the accuracy of prompt-driven activation predictions. We measure the fraction of voxels within each category-selectin region whose peak predicted activation aligns with the category indicated by the natural language query. Results confirm that BraInCoRL paired with language embeddings achieves high level of alignment between predicted voxel selectivity and query semantics across multiple subjects. The predictions for word-selective voxels notably less accurate. We hypothesize this discrepancy arises from the developmental and experiential variability inherent to the formation of word-selective regions, as these areas form predominantly through individualized learning experiences, such reading and linguistic exposure, leading to greater inter-subject variability.
Discussion
Limitations and Future Work. Here we have shown that meta-learning an in-context model of higher visual cortex can yield high performance and strong data efficiency gains, outperforming anatomical (FsAverage) and subject-wise baselines on novel subjects. Our work currently focuses on static natural images, and extensions to dynamic stimuli would likely require rethinking of the encoder backbone and network structure. Further, while we show strong generalization across scanners and utilize the largest fMRI dataset that is NSD, there may still limitations in dataset diversity [115]. Collection of larger and more diverse fMRI datasets will help mitigate this issue.
Conclusion. We introduce a foundation model that serves as an fMRI encoder, mapping from natural images voxelwise activations. We demonstrate that method adapt without any finetuning to new stimuli, subjects, scanners, and scanning protocols. Our model achieves this by meta-learning across voxels from different subjects, and performing in-context learning across stimuli. Our approach has significant data data-efficeency and performance gains over baseline methods, and has the potential help understand cortical structure data-constrained environments.
10

---

References
[1] Stephen W Kuffler. Discharge patterns and functional organization of mammalian retina. Journal of neurophysiology, 16(1):37-68, 1953.
[2] David H Hubel and Torsten N Wiesel. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. The Journal of physiology, 160(1):106, 1962.
[3] Joseph J Atick and A Norman Redlich. What does the retina know about natural scenes? Neural computation, 4(2):196-21)) 1992.
[4] Jan Verweij, Eric Hornstein, and Julie L Schnapf. Surround antagonism in macaque cone photoreceptors. Journal of Neuroscience, 23(32): (024)):024991022 10257, 2003.
[5] Kalanit Grill-Spector and Rafael Malach. The human visual cortex. Annu. Rev. Neurosci, 27(1): 649-677, 2004.
[6] Michael J Tarr and Isabel Gauthier. FFA: flexible fusiform area for subordinate-level visual processing automatized by expertise. Nature neuroscience, 3(8):764-769, 2000
[7] Isabel Gauthier, Pawel Skudlarski, John C Gore, and Adam W Anderson. Expertise for cars and birds recruits brain areas involved in face recognition. Nature neuroscience, 3(2):191-197, 2000.
[8] Roel M Willems, Marius V Peelen, and Peter Hagoort. Cerebral lateralization of face-selective and body-selective visual areas depends on handedness. Cerebral cortex, 20(7):171--1725, 2010.
[9] Qing Cai, Lise Van der Haegen, and Marc Brysbaert. Complementary hemispheric specialization for language production and visuospatial attention. Proceedings of the National Academy of Sciences, (10(4):E322-E336) 2013.
[10] Philippe Pinel, Christophe Lalanne, Thomas Bourgeron, Fabien Fauchereau, Cyril Poupon, Eric Artiges, Denis Le Bihan, Ghislaine Miiaene-LLamberte and Stanislas Dehaene. Genetic and environmental influences on the visual word form and fusiform face areas. Cerebral Cortex, 25 (9):2478-243) 2015.
[11] Zeynep M Saygin, David E Osher, Elizabeth S Norton, Deanna A Youssoufian, Sara D Beach, Jenelle Feather, Nadine Gaab, John DE Gabrieli, and Nancy Kanwisher. Connectivity precedes function in the development of the visual word form area. Nature neuroscience, 19(9):225--125 2016.
[12] Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan Solomon, Darren Seibert, and James DiCarlo. Performance-optimizee hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences, 11 111((3)88619-8)))
2014.
[13] Aria Yuan Wang, Kendrick Kay, Thomas Naselaris, Michael J Tarr, and Leila Wehbe. Better models of human high-level visual cortex emerge from natural language supervision with large and diverse dataset. Nat Mach Intell, 5:1415-1426 2023.
[14] Colin Conwell, Jacob S Prince, Kendrick N Kay, George A Alvarez, and Talia Konkle. What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines? BioRxiv, pages 2022-03 03, 2022.
[15] Zirui Chen and Michael F. Bonner. Universal dimensions of visual representation, 2024.
[16] Thomas Nasclaris, Kendrick N Kay, Shinji Nishimoto, and Jack L Gallant. Encoding and decoding fmri. Neuroimage, 56(2)400-410, 2011.
[17] Yukiyasu Kamitani and Frank Tong. Decoding the visual and subjective contents of the human brain. Nature neuroscience, 8(5):69-6685 2005.
[18] Kenneth A Norman, Sean M Polyn, Greg J Detre, and James Haxby. Beyond mind-reading: muxti-voxel pattern analysis of fmri data. Trends in cognitive sciences, 10(9):424 430, 2006.
[19] Kuan Han, Haiguang Wen, Junxing Shi, Kun-Han Lu, Yizhen Zhang, Di Fu, and Zhongming Liu. Variational autoencoder: An unsupervised model for encoding and decoding fmri activity in visual cortex. Neurolmage, 198: 136.136, 2019.

---

[20] Katja Seeliger, Umut Güçlü, Luca Ambrogioni, Yagmur Güçlütürk, and Marcel AJ van Ger- ven. Generative adversarial networks for reconstructing natural images from brain activity. Neurolmage, 181:775-785, 2018.
[21] Guohua Shen, Tomoyasu Horikawa, Kei Majima, and Yukiyasu Kamitani. Deep image recon- struction from human brain activity. PLoS computational biology, 15(1)3e0006633 2019.
[22] Ziqi Ren, Jie Li, Xuetong Xue, Xin Li, Fan Yang. Zhicheng Jiao, and Xinbo Gao. Reconstructing seen image from brain activity by visually-guided cognitive representation and adversarial learning. Neurolmage, 228: 17602, 2021.
[23] Yuqin Dai, Zhouheng Yao, Chunfeng Song, Qihao Zheng, Weijian Mai, Kunyu Peng, Shuai Lu, Wanli Ouyang, Jian Yang, and Jiamin Wu. Mindaligner: Explicit brain functional alignment for cross-subject visual decoding from limited fmri data, 2025. URL https://aaaat org/ /bbs// 2502. 05034.
[24] Alessandro T Gifford, Benjamin Lahner, Pablo Oyarzo, Aude Oliva, Gemma Roig, and Ra- doslaw M Cichy. What opportunities do large-scale visual neural datasets offer to the vision sciences community? Journal of Vision, 24(10):152-152, 2024.
[25] Serge O Dumoulin and Brian A Wandell. Population receptive field estimates in human visual cortex. Neuroimage, 39(2):647-66) 2008
[26] Umut Güçlü and Marcel AJ Van Gerven. Deep neural networks reveal gradient in the complexity of neural representations across the ventral stream. Journal of Neuroscience, 35(77); 10005-10014 2015.
[27] David Klindt, Alexander S Ecker. Thomas Euler, and Matthias Bethge. Neural system identi- fication for large populations separating "what" and "where". Advances in neural information processing systems, 30, 2017.
[28] Michael Eickenberg, Alexandre Gramfort, Gaël Varoquaux, and Bertrand Thirion. Seeing it all: Convolutional network layers map the function of the human visual system. Neurolmage, 152: 184-194, 2017.
[29] Haiguang Wen, Junxing Shi, Yizhen Zhang, Kun-Han Lu, Jiayue Cao, and Zhongming Liu. Neural encoding and decoding with deep learning for dynamic natural vision. Cerebral cortex, 28(12):413-4466) 2018
[30] Guy Gaziv, Roman Beliy, Niv Granot, Assaf Hoogi, Francesca Strappini, Tal Golan, and Michal Irani. Self-supervised natural image reconstruction and large-scale semantic classification from brain activity. Neurolmage, 191211, 2022.
[31] Hossein Adeli, Sun Minni, and Nikolaus Kriegeskorte. Predicting brain activity using trans- formers. bioRxiv, pages 2023-08, 2023.
[32] Guangyin Bao, Qi Zhang, Zixuan Gong, Zhuojia Wu, and Duoqian Miao. Mindsimulator: Exploring brain concept localization via synthetic fmri. arXiv preprint arXiv:2503.02351, 2025.
[33] Roman Beliy, Navve Wasserman, Amit Zalcher, and Michal Irani. The wisdom of a crowd of brains: A universal brain encoder. arXiv preprint arXiv:2406. 12179, 2024.
[34] Mcenakshi Khosla and Leila Wehbe. High-level visual areas act like domain general filters with strong selectivity and functional specialization. bioRxiv, pages 2022-03, 2022.
[35] Meenakshi Khosla, Keith Jamison, Amy Kuceyeski, and Mert Sabuncu. Characterizing the ventral visual stream with cesponse-pptmizze neural encoding models. Advances in Neural Information Processing Systems, 35:9389- 9402,4402220 2022.
[36] Cory Efird, Alex Murphy, Joel Zylberberg, and Alona Fyshe. What's the opposite of a face? finding shared decodable concepts and their negations in the brain. arXiv e-prints, pages arXiv- 2405, 2024.
[37] Huzheng Yang, James Gee, and Jianbo Shi. Brain decodes deep nets. Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 23030-23040, 2024.
12

---

[38] Huzheng Yang, James Gee, and Jianbo Shi. Alignedcut: Visual concepts discovery on brain- guided universal feature space. arXiv preprint (rXiv:240.183444 2024.
[39] Andrew F Luo, Jacob Yeung, Rushikesh Zawar, Shaurya Dewan, Margaret M Henderson, Leila Wehbe, and Michael J Tarr. Brain mapping with dense features: Grounding cortical semantic selectivity in natural images with vision transformers. arXiv preprint arXiv:2410.0526. 2024.
[40] Gabriel H. Sarch, Michael J. Tarr, Katerina Fragkiadaki, and Leila Wehbe. Brain dissection: fmri-trained networks reveal spatial selectivity in the processing of natural images. bioRxiv, 2023. doi: 10. 1011100/003300 0...........2335. URL https: /www. biorxiv. wrgg arg.conteet//arlyyy. 2023/11/20/2023 05 29. 542635.
[41] Alexander I. Lappe, Anna Bognár, Ghazaleh Ghamkahri Nejad, Albert Mukovskiy, Lucas Martini, Martin Giese, and Rufin Vogels. Parallel backpropagation for stared-feature visualization. Advances in Neural Information Processing Systems, 37:22993-23012, 2024.
[42] Edgar Y Walker, Fabian H Sinz, Erick Cobos, Taliah Muhammad, Emmanouil Froudarakis, Paul Fahey, Alexander S Ecker, Jacob Reimer, Xaq Pitkow, and Andreas Tolias. Inception loops discover what excites neurons most using deep predictive models. Nature neuroscience, 22 (12):2060-2065, 2019.
[43] Pouya Bashivan, Kohitij Kar, and James DiCarlo. Neural population control via deep image synthesis. Science, (6336649) 2019.
[44] Carlos R Ponce, Will Xiao, Peter F Schade, Till S Hartmann, Gabriel Kreiman, and Margaret S Livingstone. Evolving images for visual neurons using a deep generative network reveals coding principles and neuronal preferences. Cell, 177(4))999-1000) 2019.
[45] N Apurva Ratan Murty, Pouya Bashivan, Alex Abate, James DiCarlo, and Nancy Kanwisher. Computational models of category brain regions enable high-throughput tests of selec- tivity. Nature communications, 12(1):5540, 2021.
[46] Zijin Gu, Keith Wakefield Jamison, Meenakshi Khosla, Emily Allen, Yihan Wu, Ghislain St- Yve,, Thomas Naselaris, Kendrick Kay, Mert R Sabuncu, and Amy Kuceyeski. NeuroGen: activation optimized image synthesis for discovery neuroscience. Neurolmage, 247:118812, 2022.
[47] Pawel A Pierzchlewicz, Konstantin Willeke, Arne Nix, Pavithra Flumalai, Kelli Restivo, Tori Shinn, Cate Nealley, Gabrielle Rodriguez. Saumil Patel, Katrin Franke, et al. Energy guided diffusion for generating neurally exciting images, 2023.
[48] Andrew F Luo, Margaret M Henderson, Leila Wehbe, and Michael J Tarr. Brain diffusion for visual exploration: Cortical discovery using large scale generative models. arXiv preprint arXiv:2306.0308) 2023.
[49] Diego Gareía Cerdas, Christina Sartzetaki, Magnus Petersen, Gemma Roig, Pascal Mettes, and Iris Groen. Brainactiv: Identifying visuo-semantic properties driving cortical selectivity using cuffusion-basee image manipulation. bioRxiv, pages 1024-10, 2024.
[50] Andrew Luo, Margaret Marie Henderson, Michael J. Tarr, and Leila Wehbe. Brainscuba: Fine-grained natural language captions of visual cortex selectivity. The Twelfth International Conference on Learning Representations, 2024. URL
net/forum?id
mQYHXUUTKU.
[51] Takuya Matsuyama, Shinji Nishimoto, and Yu Takagi. Lavea: Llm-assisted visual cortex captioning. arXiv preprint 2oXiv:2502.13600 2025.
[52] Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. bioRxiv, pages 2022-11, 2022.
[53] Zijiao Chen, Jiaxin Qing, Tiange Xiang, Wan Lin Yue, and Juan Helen Zhou. Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding. In Proceedings of the IEFF/CVF Conference on Computer Vision and Pattern Recognition, pages 227 10002020,22222221111 2023.
13

---

[54] Yizhuo Lu, Changde Du, Dianpeng Wang, and Huiguang He. Minddiffuser: Controlled image reconstruction from human brain activity with semantic and structural diffusion. arXiv preprint arXiv:2303. /4139, 2023.
[55] Furkan Ozcelik and Rufin VanRullen. Brain-diffuser: Natural scene reconstruction from fmri signals using generative latent diffusion. arXiv preprint arXiv:2303.0533:4 2023.
[56] Adrien Doerig, Tim C Kietzmann, Emily Allen, Yihan Wu, Thomas Naselaris, Kendrick Kay, and Ian Charest. Semantic scene descriptions as an objective of human vision. arXiv preprint arXiv:2209. / 1737, 2022.
[57] Matteo Ferrante, Furkan Ozcelik, Tommaso Boccato, Rufin VanRullen, and Nicola Toschi. Brain captioning: Decoding human brain activity into images and text. arXiv preprint arXiv:23305. /560, 2023.
[58] Yulong Liu, Yongqiang Ma, Wei Zhou, Guibo Zhu, and Namning Zheng. Brainelip: Bridging brain and visual- visual-linguistic representation via clip for generic natural visual stimulus decoding from fmri. arXiv preprint arXiv:2302. /2971, 2023.
[59] Weijian Mai and Zhijun Zhang. Unibrain: Unify image reconstruction and captioning all in one diffusion model from human brain activity. arXiv preprint arXiv:2308.07428, 2023.
[60] Paul S Scotti, Mihir Tripathy, Cesar Kadir Torrico Villanueva, Reese Kneeland, Tong Chen, Ashutosh Narang, Charan Santhirasegaran, Jonathan Xu, Thomas Naselaris, Kenneth A Norman, et al. Mindeye2: Shared-subject models enable fmri-to-image with hour of data. arXiv preprint arXiv:2403. /207, 2024.
[61] Yohann Benchetrit, Hubert Banville, and Jean-Rémi King. Brain decoding: toward real-time reconstruction of visual perception. arXiv preprint arXiv:2310.1981 2023.
[62] Dongyang Li, Chen Wei, Shiying Li, Jiachen Zou, Haoyang Qin, and Quanying Liu. Vi- sual decoding and reconstruction via eeg embeddings with guided diffusion. arXiv preprint arXiv:2403.07721, 2024.
[63] Zhanqiang Guo, Jiamin Wu, Yonghao Song, Jiahui Bu, Weijian Mai, Qihao Zheng, Wanli Ouyang, and Chunfeng Song. Neuro-3d: Towards 3d visual decoding from eeg signals, 2024. URL https: /ttp:////axvvvvvo http:/iwworg/bb///////2 abs//41s./111... 12248.
[64] Yu Zhu, Bo Lei, Chunfeng Song, Wanli Ouyang, Shan Yu, and Tiejun Huang. Multi-modal latent variables for cross-individual primary visual cortex modeling and analysis, 2024. URL
http::///xxvvvv../bbb//o www.o/ggg//ba/aa/eeeeee//eee/e/e ab4/242....22 2412. 14536.
[65] Steffen Schneider, Jin Hwa Lee, and Mackenzie Weygandt Mathis. Learnable latent embeddings for joint behavioural and neural analysis. Nature, 617(7960):30-3366 2023.
[66] Zijiao Chen, Jiaxin Qing, and Juan Helen Zhou. Cinematic mindscapes: High-quality video reconstruction from brain activity. Advances in Neural Information Processing Systems, 36: 24841-248535 2023.
[67] Zixuan Gong, Guangyin Bao, Qi Zhang, Zhongwei Wan, Duoqian Miao, Shoujin Wang, Lei Zhu, Changwei Wang, Rongtao Xu, Liang Hu, Ke Liu, and Yu Zhang. Neuroclips: Towards high-fidelity and smooth fMRI-to-video reconstruction. In The Thirty-eighth Annual Conference on Neural Information Processing Systems, 2024. URL https: / //tp///pppe/eeeeeeeeeeeeeeeeeeee net/foruumt
id-8qu52F11Dt.
[68] Jacob Yeung, Andrew Luo, Gabriel Sarch, Margaret M Henderson, Deva Ramanan, and Michael J Tarr. Neural representations of dynamic visual stimuli. arXiv preprint arXiv:2406.02655, 2024.
[69] Xuan-Hao Liu, Yan-Kai Liu, Yansen Wang, Kan Ren, Hanwen Shi, Zilong Wang, Dongsheng Li, Bio-Liang Lu, and Wei-Long Zheng. Feg2video: Towards decoding dynamic visual perception from eeg signals. Advances in Neural Information Processing Systems, 37:72245-72273, 2024.
14

---

[70] Camilo Fosco, Benjamin Lahner, Bowen Pan, Alex Andonian, Emilie Josephs. Alex Lascelles, and Aude Oliva. Brain netflix: Scaling data reconstruct videos from brain signals. In European Conference on Computer Vixion, pages 457-474, Springer, 2024.
[71] Brian N Pasley, Stephen V David, Nima Mesgarani, Adeen Flinker, Shihab A Shamma, Nathan E Crone, Robert Knight, and Edward F Chang. Reconstructing speech from human auditory cortex. PLoS biology, 1001251, 2012.
[72] Gaël Varoquaux, Pradeep Reddy Raamana, Denis A Engemann, Andrés Hoyos-Idrobo, Yannick Schwartz, and Bertrand Thirion. Assessing and tuning brain decoders: cross-validation, caveats, and guidelines. Newrolmage, 145:166-111 179, 2017.
[73] Ludovic Bellier, Anaïs LJorens, Déborah Marciano, Aysegul Gunduz, Gerwin Schalk, Peter Brumner, and Robert T Knight. Music can be reconstructed from human auditory cortex activity using nonlinear decoding models. PLoS biology, 2108):33021176 2023.
[74] Subba Reddy Oota, Emin Çelik, Fatma Deniz, and Mariya Toneva. Speech language models lack important brain-relevant semanties. arXiv preprini arXiv:231 1..311..00 1.04664, 2023.
[75] Hyejeong Jo, Yiqian Yang. Juhyeok Han, Yiqun Duan, Hui Xiong, and Won Hee Lee. Are eeg-to-text models working? arXiv preprint arXiv:240S.0ms 22455.66999...... 2024.
[76] Francis R Willett, Erin M Kunz, Chaofei Fan, Donald T Avansino, Guy H Wilson, Eun Young Choi, Foram Kamdar, Matthew F Glasser, Leigh R Hochberg, Shaul Druckmann, et al. A high-periin perhfoerforaacc speech neuroprosthesis. Nature, (0079976)103111000 2023.
[77] Sean L Metzger, Kaylo Littlejohn, Alexander Silva, David Moses, Margaret Seaton, Ran Wang, Maximilian E Dougherty, Jessie R Liu, Peter Wu, Michael A Berger, al. A high-performance neuroprosthesis for speech decoding and avatar control. Nature, 620(7976): 1037-1046, 2023.
[78] Justine Sergent, Shinsuke Ohta, and Brennan Macdonald. Functional neuroanatomy of face and object processing: a positron emission tomography study. Brain, 115(1):15-36, 1992.
[79] Truett Allison, Gregory McCarthy, Anna Nobre, Aina Puce, and Aysenil Belger. Human extrastriate visual cortex and the perception of faces, words, numbers, and colors. Cerebral cortex, 4(5):544-554, 1994.
[80] Gregory McCarthy, Aina Puce, John C Gore, and Truett Allison. Face-specific processing in the human fusiform gyrus. Journal of cognitive neuroscience, 955):05-6610 1997.
[81] Nancy Kanwisher, Josh McDermott, and Marvin M Chun. The fusiform face area: a module in human extrastriate cortex specialized for face perception. Journal of neuroscience. 17(11): 1997.
[82] Geoffrey K Aguirre, John A Detre, David C Alsop, and Mark D' E'sposito. The parahippocampus subserves topographical learning in man. Cerebral cortex, 6(6):223-82, 1996.
[83] Russell Epstein and Nancy Kanwisher. A cortical representation of the local visual environment. Nature, (926667)) 1998.
[84] Paul E Downing, Yuhong Jiang, Miles Shuman, and Nancy Kanwisher, A cortical area selective for visual processing of the human body. Science, 993(5539):2470-2477) 2001.
[85] Kalanit Grict-Spector. The neural basis of object perception. Current opinion in neurobiology. 159-111 2003.
[86] Rafael Malach, JB Reppas, RR Benson, KK Kwong, H Jiang, WA Kennedy, PJ Ledden, TJ Brady, BR Rosen, and RB Tootell. Object-related activity revealed by functional magnetic resonance imaging in human occipital cortex. Proceedings of the National Academy of Sciences, (2(18)81135-8139 1995.
[87] Meenakshi Khosla, N. Apurva Ratan Murty, and Nancy Kanwisher. A highly selective response to food in human visual cortex revealed by hypothesis-free free voxel decomposition. Current Biology, 32:1-13, 2022.
15

---

[88] Ian ML Pennock, Chris Racey, Emily J Allen, Yihan Wu, Thomas Naselaris, Kendrick N Kay, Anna Franklin, and Jenny M Bosten. Color-biased regions in the ventral visual pathway are food selective. Current Biology, 33(1):134-144) 146, 2023.
[89] Nidhi Jain, Aria Wang, Margaret M. Henderson, Ruogu Lin, Jacob S. Prince, Michael J. Tarr, and Leila Wehhe. Selectivity for food in human ventral visual cortex. Communications Biology doi: 00000//4440000230444444444
2023 6:1, 6:1-14, 2 2023. ISSN 2399-3642.
[90] Laurent Cohen, Stanislas Dehaene, Lionel Naccache, Stéphane Lehéricy, Ghislaine Dehaene- Lambertz, Marie-Anne Hénaff, and François Michel. The visual word form area: spatial and temporal characterization of initial stage of reading in normal subjects and posterior split-brain patients. Brain, 123(2):291-377) 2000.
[91] Stanislas Dehaene, Lionel Naccache, Laurent Cohen, Denis Le Bihan, -aan-François Mangin, Jean-Baptin Baptiste Poline, and Denis Rivière. Cerebral mechanisms of word masking and unconscious repetition priming. Nature neuroxcience, 4(7):752-758, 2001.
[92] Hans P Op de Beeck, Jennifer A Deutsch, Wim Vanduffel, Nancy G Kanwisher, and James J DiCarlo. A stable topography of selectivity for unfamiliar shape classes in monkey inferior temporal cortex. Cerebral Cortex, 1877):167--164, 2008.
[93] Job van den Hurk, Mare Van Baelen, and Hans P Op de Beeck. Development of visual category selectivity in ventral visual cortex does not require visual experience. Proceedings of the National Academy of Sciences, 114(22):E4501-EF
2017.
[94] Golijeh Golarai, Alina Liberman, and Kalanit Grill-Spector. Experience shapes the development of neural substrates of face processing in human ventral temporal cortex. Cerebral Cortex, 27(2): Ahv 2015.
[95] Tina T Liu, Adrian Nestor, Mark D Vida, John A Pyles, Christina Patterson, Ying Yang, Fan Nils Yang, Erez Freud, and Marlene Behrmann. Successful reorganization of category-selective visual cortex following occipito-temporal lobectomy in childhood. Cell reports, 24(5):1133-1122 2018.
[96] Hans P Op de Beeck, Ineke Pillet, and J Brendan Ritchie. Factors determining where category- selective areas emerge visual cortex. Trends in cognitive sciences, 23(9):784-797, 2019.
[97] Nooshin Abbasi, John Duncan, and Reza Rajimehr. Genetic influence is linked to cortical morphology in category-selective areas of visual cortex. Nature Communications, 11(1):709, 2020.
[98] Alfonso Nieto-Castanón and Evelina Fedorenko. Subject-specific functional localizers increase sensitivity and functional resolution of multi-subject analyses. Neuroimage, 6363)16646-1669, 2012.
[99] Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adap- tation of deep networks. In International conference machine learning, pages 1126-1135. PMLR, 2017.
[100] Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02991 2018
[101] Aravind Rajeswaran, Chelsea Finn, Sham Kakade, and Sergey Levine. Meta-learning with implicit gradients. Advances in neural information processing systems, 32, 2019.
[102] Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems, 30, 2017.
[103] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
16

---

[104] Johannes Von Oswald. Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning, pages 35151-35174. PMIR, 2023.
[105] Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. Metaicl: learn in context. arXiv preprint arXiv:2110.1 2021.
Learning to
[106] Julian Coda-Forno, Marcel Binz, Zeynep Akata, Matt Botvinick, Jane Wang, and Eric Schulz. Meta-in-context learning in large language models. Advances in Neural Information Processing Systems, 36:65189-652222 89-662001 2023.
[107] Shivam Garg, Dimitris Tsipras, Percy Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems, 35:30583-30553 305988, 2022.
[108] Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can gpt learn in-context? language models implicitly perform gradient descent meta-optimizers. arXiv preprint wrxiv2222..10559, 2022.
[109] Jianlin Su. Analyzing the scale operation of attention from the perspective of entropy invariance, Dec 2021. URL https:tt///. /kexue fm/ http/rrcciees////eeee /8823.
[110] David Chiang and Peter Cholak. Overcoming theoretical limitation of self-attention. arXiv preprint arXiv:2202. 12172, 2022.
[111] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16660 2023.
[112] Emily Allen, Ghislain St-Yves, Yihan Wu, Jesse L Breedlove, Jacob S Prince, Logan Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, et al. A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence. Nature neuroscience, 2661):1 2022.
[113] Nadine Chang, John A Pyles, Austin Marcus, Abhinav Gupta, Michael J Tarr, and Elissa M Aminoff. Bold5000, a public fMRI dataset while viewing 5000 visual images. Scientific Data, (1):1-18, 2019.
[114] Colin Conwell, Jacob Prince, Kendrick N Kay, George A Alvarez, and Talia Konkle. A large-scale examination of inductive biases shaping high-level visual representation in brains and machines. Nature communications, 15(1):9383, 2024.
[115] Ken Shirakawa, Yoshihiro Nagano, Misato Tanaka, Shuntaro C Aoki, Kei Majima, Yusuke Muraki, and Yukiyasu Kamitani. Spurious reconstruction from brain activity. arXiv preprint AXiv:2405.10077, 2024.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The claims made in the abstract and introduction are reflected and addressed in the experiments section.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitation of the work is discussed in the discussion section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
18

---

Justification:
The paper does not include theoretical results.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proof' can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
The learning theory is described in the methods section. The model training details are discussed the experiment setup section and in the supplemental material section. We will release the code and checkpoints before the meeting.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of large language model), releasing of a model checkpoint, or other means that appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) the contribution is primarily a new algorithm, the paper should make clear how reproduce that algorithm. (b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
19

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification: All the data, code and checkpoints will be published before the meeting (with GitHub link included). Also, the instructions to reproduce experimental results are provided in the experimental results and supplemental material sections.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (uttps://nips.c cc/
public/gue wwwps//ooeSSuuueseeeeeeooo
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
//wpps.cc///////// httppscccppblccccccecceeecc htt//uudessss Cooeeubmiiss onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The major experimental settings including how to train and test the model is described in the experiments section. More details are provided in the supplemental material section.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can provided either with the code, appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification: The experiment results are accompanied by error bars in the experiments section and supplemental material section when available.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoor bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
20

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative error rates).
If error bars reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: The details on computational resources are specified in the supplemental material section.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental rums as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't iidn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
http://uuurrpp.ccppup..pee cctppblic////cccccccccccccccccct Beteee/thiiihheeeeeeeeee csGuidelines
Answer: [Yes]
Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The points are discussed in the discussion section and supplemental material section.
Guidelines:
21

---

The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negatíve societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (c.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of MI.).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
To the best of our knowledge, the paper poses no such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: The creators or original owners of assets are properly credited and the license and terms of use are explicitly mentioned and properly respected in the main text and in the supplemental material.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
22

---

For scraped data from a particular source (e.g., website). the copyright and terms of service of that souree should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode com datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is available online, the authors encouraged to reach out the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification:
We release model's code via our anonymous GitHub URL.
Guidelines:
The answer NA means that the paper does not release new assets. Researchers should communicate the details of the dataset/code/model part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, as well as details about compensation (if any)?
Question:
Answer: [NA]
Justification: No new data collection or crowdsourcing was performed. Our analyses use only the public BOLD5000 and NSD dataset.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurlPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: We did not collect any new human-subject data. All analyses are performed on the publicly available BOLD5000 and NSD fMRI datasets, which require no additional IRB review.
Guidelines:
23

---

The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: The core method development in this research does involve LLMs as any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to LLM policy (https: / www.c///eee/ee//////////2// /LLM) for what should or should not be described.
24

---

A Technical Appendices and Supplementary Material
Sections
1. Social impacts (Section A.1)
2. Implementation details (Section A.2)
3. Text prompts for category-selectiv regions (Section A.3)
4. A more detailed description of logit scaling (Section A.4)
5. Cortex prediction explained variance for different image encoding backhones and subjects (Section A.5)
6. Voxelwise performance across five category-selectiv regions for different image encoding backbones and subjects (Section A.6)
7. Voxelwise explained variance across varying support set sizes for more subjects, backbones and for pretrain-only models (Section A.7)
8. Impact of holding out the test subject's unique images during meta-training evaluated on more backbones (Section A.8)
9. Correlation of each backbone's predictions with fully trained activation predictions (Sec-
tion A.9)
10. Voxelwise explained variance evaluation in BOLD5000 for more subjects and different backbones (Section A.)
11. Dimensional reduction of predicted response function weights on more subjects (Sec-
tion A.11)
12. Predicting cortical responses from natural language prompts on more subjects (Section A.12)
13. Voxelwise prompt classification accuracy for more subjects (Section A.13)
14. Additional evaluation of BrainCoRL on NSD dataset (Section A.
15. Additional evaluation of BraInCoRL on BOLD5000 dataset (Section A.15)
16. Evaluation of each training stage's contribution (Section A.16)
17. Performance of BraInCoRL conditioned on the full 9000-image set (Section A.17)
18. Evaluation the choice of loss function during training (Section A.18)
19. Ablation on logit scaling (Section 19)
25

---

A.1 Social impacts
Our work introduces BraInCoRL, a meta-learning framework that uses IMRI-measured voxel re- sponses and trained visual encoders to perform in-context adaptation: given a small support set of image-response pairs, the model directly estimates each voxel's 's response- funontiin parameters for novel stimuli. BraInCoRI''s fusion of diverse image data and neural measurements uncovers data-driven principles of cortical organization heyond both traditional hypothesis-driven experiments and computational encoding models that require thousands of samples. Moreover, its alignment of neural responses with natural language prompts enables the generation of new hypotheses about semantic representation in visual cortex. As such, BraInCoRL may accelerate early diagnosis and monitoring visual or neurological disorders via rapid, subject-specific cortical mapping; guide more efficient experimental design through optimized stimulus selection; deepen our understanding of se- mantic coding and inter-subject variability; and, when integrated with generative image models, open avenues for brain- maaide stimulus synthesis and personalized neuroprosthetic and brain-compaan computer interface development. While BraInCoRL offers significant potential for neuroscience and clinical applications, its reliance fMRI datasets and computational infrastructure may limit accessibility for under- resourced research groups and raise privacy concerns if applied to sensitive neural data.
A.2 Implementation details
Network architecture.
Our BraInCoRL model architecture comprises three main components:
1. Input projection. An input projection MLP layer is applied to each token individually, which maps the stimulus semantics and voxel activation into an embedding. In detail, we concatenate each image embedding with its scalar neural response and pass it through single-layer MLP to align the two modalities into a unified internal feature space.
2. Transformer encoder. A stack of 20 self-attention layers integrates information across all support examples (plus learnable tokens), capturing contextual relationships and the relative importance of each in-context sample. We adopt best practices and utilize SwiGLU activation paired with pre-normalization in the attention block. We utilize 10 heads in each multi-head self attention.
3. Weight prediction. The [CLS] token from the final layer goes through an MLP to output a hyperweight which is used to parameterize the final encoder. In detail, the aggregated representation is fed through another single-layer MLP that outputs a weight vector, which is then used to linearly combine unknown-image embeddings to produce the final neural response predictions.
The CLIP-based variant (encoding dimension E = 512) contains approximately 97.2 M parameters; DINO (E' = 768) and SIGLIP (E 1152) variants comprise roughly 112 M and 130 M parameters, respectively.
Model training. Training is implemented in PyTorch eight NVIDIA RTX 6000 Ada GPUs (48 GB each). We optimize using A (decoupled weight decay x 10 4) with an initial learning rate of 1 x 10-3 which decays to 1 x 10 via ReduteeLROnPlaeauu scheduler (factor 0.1, patience 5 epochs, cooldown epochs, threshold 1c-4). Mini- batches randomly sample an in-context support set of 100 images in the first pretraining stage, and randomly sample between 30 and 500 in-context support images in the second context extension stage and the third finetuning stages. Each training stage runs for up to 100 epochs with early stopping based on validation loss (patience: 5 epochs). The training batch size is 80. We allocate 20% of the test set data for validation.
Computational cost. With an in-context support of 100 images, our model predicts responses for ~ 20,000 voxels in the higher visual cortex in under 20 seconds on a single RTX 6000 Ada GPU.
26

---

A.3 Text prompts for category-selective regions
We define a natural language prompts for each semantic category. For every maage-prompl pair, we use the CLIP text encoder to generate text encodings. The natural language prompts for each category are listed below:
Faces [A photo a person's face, portrait photo of face, face facing the camera, A photo of face, A photo of a human face, photo of faces, People looking at the camera, portrait of person, A portrait photo]
Bodies [A photo of a torso, photo of limbs, A photo of bodies, A photo of person, A photo of people, A photo of body, A person's arms, A person's s legs, A photo of hands]
Places
photo of a bedroom, photo of an office, photo of a hallway, A photo of doorway, A photo of interior design, A photo of a building, photo of house, photo of nature, photo of a landscape]
Food [A photo of food, A photo of cuisine, A photo of fruit, A photo of meal, photo bread, photo of rice, photo of snack, A photo of pastries]
foodstuffs, photo
Words [A photo of words, A photo of glyphs, A photo of a glyph, A photo of text, A photo of numbers, A photo of letter, photo of letters, A photo of writing, A photo of text on an object]
27

---

A.4 A more detailed description of logit scaling
The motivating factor underlying logit scaling our desire have our in-context learned encoder perform well regardless of the number of stimuli given to the model, and effectively generalize to context sizes beyond those seen during training. For example, while we may only train with between 30 and 500 images, a third-party experimenter may want to use fewer than 30 images or more than 500 images to condition the model. Across all cases, we want the model succeed.
This logit scaling method was first proposed in [l. 2]. and later adapted in Qwen LLM (logn-scaling) [3] and Llama 4 LLM (temperature scaling) [4]. We will briefly summarize the high-level math, which we take from [1] with our commentary:
Let the attention value in self-attention for particular query token i to value token be
Aq k;
j
eq
Then the entropy is defined
H, =
ai log asj
==1
Substituting the expression for as,y we have the entropy as
e Ag.k
H4 - Σ ais log
Σラ=1 edg kj
Since Σj=1 ai 1, we can express the formula as
Σ Ag ki
(Aq kj)
H4 = log
bq k;
5=l
Σ ed kk
If the first term is expressed as an expectation over we have
Σ bq k;
sacks k.
Which leads the following upproximation of entropy
AB le ky (qi k2)]
H4 R log n - log E, [c k3]
the vectors are assumed to the output of a layernorm layer with length /d, the expectation can be converted to over the angles between vectors
AdIE (e cos
Hi 2J log + log Eg Ads 01-
Eg [ Ad
Since most randomly distributed vectors in higher dimensions are orthogonal, we derive a term which can roughly be expressed as
H, R log C
28

---

where C' does not depend on the number of tokens n.
This leads to approximate scaling factor for the logits of log keep the entropy invariant to context length.
Therefore, we change the standard formulation of attention values by applying scaling factor of log n to the QK term, as shown Equation (4) of our paper.
Note that in the above explanation we adopt the notation from [[].
29

---

A.5 Cortex prediction explained variance for different image encoding backbones and subjects
In this section, we compare three methods across multiple subjects (S1-SB) and embedding backbones (CLIP, DINO, SigLIP): the fully trained reference model fit converge on each subject's 's full 9,000- image training set; our BraInCoRI. approoch,, which adapts to a new subject with only 100 in-context images; and within-subject ridge regression baseline trained on the same 100 images with the BraInCoRI. approach. In every case, BraInCoRI. outperforms ridge regression and achieves accuracy similar that of the fully trained model.
0.0
0.7
Subji
Subj2
Subj3
Subj4
Fully Trained
BrainCoRL
Ridge Regression
Figure S.1: Higher visual cortex explained variance of CLIP backhone. From left right, show the explained variance from the full model trained on 9000 images for a subject, BraInCoRL with just 100 in-context images from the new subject, and within-subject ridge regression with 100 images using CLIP backbone for subject 1,2,3,4.
30

---

0.0
0.7
Subj5
 ooodot JUUU
Subj6
Subj7
Subj8
Fully Trained
BrainCoRL
Ridge Regression
Figure S.2: Higher visual cortex explained variance of CLIP backbone. From left right, show the explained variance from the full model trained on 9000 images for subject, BraInCoRL with just 100 in-context images from the new subject, and within-subject ridge regression with 100 images using CLIP backbone for subject 5, 6, 7, 8.
31

---

0.0
0.7
Subj1
Subj2
Subj3
Subj4
Fully Trained
BrainCoRL
Ridge Regression
Figure S.3: Higher visual cortex explained variance of DINO backbone. From left right, show the explained variance from the full model trained on 9000 images for subject, BraInCoRL with just 100 in-context images from the new subject, and within-subjer ridge regression with 100 images using DINO backbone for subject 1, 2, 3, 4.
32

---

0.0
0.7
Subj5
Subj6
Subj7
Subj8
Fully Trained
BrainCoRL
Ridge Regression
Figure S.4: Higher visual cortex explained variance of DINO backbone. From left right,
show the explained variance from the full model trained on 9000 images for subject, BraInCoRL with just 100 in-context images from the new subject, and within-subjer ridge regression with 100 images using DINO backbone for subject 5, 6, 7, 8.
33

---

0.0
0.7
Subji
DUDDE
Subj2
Subj3
Subj4
Fully Trained
BrainCoRL
Ridge Regression
Figure S.5: Higher visual cortex explained variance of SigLIP backbone. From left right, show the explained variance from the full model trained on 9000 images for subject, BraInCoRL ridge regression with 100
with just 100 in-context images from the new subject, and images using SigLIP backbone for subject 1, 2, 3, 4.
34

---

0.0
0.7
Subj5
Subj6
Subj7
Subj8
Fully Trained
BrainCoRL
Ridge Regression
Figure S.6: Higher visual cortex explained variance of SigLIP backbone. From left right,
show the explained variance from the full model trained on 9000 images for subject, BraInCoRL ridge regression with 100
with just 100 in-context images from the new subject, and images using SigLIP backbone for subject 5, 6, 7, 8.
35

---

A.6 Voxelwise performance across five category-selective regions for different image encoding backbones and subjects
In this section, we report voxel-wise explained variance in five category regions (faces, places, bodies, words, and food) with CLIP, DINO and SigLIP backbone for subjects S1-S8. We compare our in-context model (BraInCoRL.) against the fully trained reference model fit to converge on each subject's full 9, 9,000-image training set, within-subject ridge regression baselines trained on 100 and 300 images, and the FsAverage map. BraInCoRL. outperforms the ridge baselines and closely approaches the performance of the fully trained model.
Table S.I: Voxel-wise explained variance with the CLIP backbone for Subjects and 2. We report performance for our in-context model (BraInCoRL), the fully trained reference ("Fully Trained'), within-subject ridge regression baselines (100, 300), and the FsAverage map across five category- selective regions (faces, places, bodies, words, food).
Faces
Places
Bodies
Words
Food
Mean
S1
S2
Si
S2
S1
S2
S2
S1
S2
Fully Trained
0.19
0.16
0.20
0.27
0.28
0.24
0.11
0.11
0.16
0.17
0.18
0.19
Ridge-100
0.10
0.07
0.08
0.14
0.16
0.12
0.02
0.03
0.05
0.07
0.07
0.08
Ridge-300
0.13
0.10
0.13
0.20
0.22
0.16
0.06
0.06
0.10
0.11
0.11
0.12
FsAverage map
0.13
0.06
0.11
0.19
0.09
0.08
0.06
0.03
0.14
0.18
0.08
0.06
BraInCoRL-100
0.16
0.13
0.16
0.23
0.25
0.21
0.07
0.08
0.12
0.13
0.13
0.15
Table S.2: Voxel-wise explained variance with the CLIP backbone for Subjects and 4. We report performance for our in-context model (BraInCoRL), the fully trained reference ("Fully Trained"), within--uubjet ridge regression baselines (100, 300), and the FsAverage map across five category- selective regions (faces, places, bodies, words, food).
Faces
Places
Bodics
Words
Food
Mcan
$3
S4
$4
S3
84
$4
S3
S3
S4
Fully Trained
0.16
0.14
0.16
0.16
0.17
0.17
0.09
0.07
0.10
0.12
0.13
0.14
Ridge-100
0.07
0.05
0.08
0.05
0.08
0.08
0.02
0.01
0.03
0.04
0.05
0.05
Ridge-300
0.10
0.09
0.11
0.10
0.11
0.12
0.04
0.04
0.05
0.07
0.08
0.09
FsAverage map
0.10
0.03
0.14
0.05
0.11
0.06
0.07
0.03
0.10
0.07
0.10
0.04
BraInCoRL-100
0.12
0.10
0.13
0.13
0.14
0.13
0.05
0.04
0.07
0.08
0.10
0.10
Table S.3: Voxel-wise explained variance with the CLIP backbone for Subjects and 6. We report performance for our in-context model (BraInCoRL), the fully trained reference ("Fully Trained"), within-subject ridge regression baselines (100, 300), and the FsAverage map across five category- selective regions (faces, places, bodies, words, food).
Faces
Places
Bodies
Words
Food
Mcan
S5
S6
S5
S6
S5
S5
S6
Fully Trained
0.24
0.17
0.32
0.13
0.26
0.18
0.17
0.09
0.24
0.09
0.23
0.11
Ridge-100
0.11
0.07
0.16
0.03
0.13
0.09
0.06
0.01
0.11
0.02
0.10
0.03
Ridge-300
0.16
0.11
0.24
0.07
0.19
0.13
0.10
0.04
0.16
0.04
0.15
0.06
FsAverage map
0.07
0.05
0.11
0.08
0.06
0.04
0.05
0.06
0.08
0.04
0.07
0.05
BraInCoRL-100
0.20
0.14
0.29
0.10
0.23
0.15
0.12
0.05
0.19
0.05
0.19
0.08
36

---

Table S.4: Voxel-wise explained variance with the CLIP backbone for Subjects and 8. We report performance for in-context model (BraInCoRL), the fully trained oracle ("Fully Trained"), within- subject ridge regression baselines (100, 300), and the FsAverage map across five category-selective regions (faces, places, bodies, words, food).
Faces
Places
Bodies
Words
Food
Mean
S7
S8
S7
S8
S7
S8
S7
S8
S7
S8
S7
S8
Fully Trained
0.14
0.08
0.18
0.10
0.21
0.09
0.12
0.04
0.12
0.06
0.16
0.09
Ridge-100
0.06
0.03
0.08
0.04
0.11
0.04
0.03
0.00
0.03
0.02
0.06
0.03
Ridge-300
0.09
0.05
0.12
0.06
0.15
0.05
0.06
0.01
0.06
0.03
0.09
0.05
PsAAerage map
0.12
0.04
0.17
0.04
0.10
0.03
0.09
0.03
0.19
0.02
0.09
0.03
BraInCoRI--100
0.11
0.07
0.15
0.08
0.18
0.07
0.08
0.02
0.08
0.04
0.12
0.07
Table S.5: Voxel-wise explained variance with the DINO backbone for Subjects and 2. We report performance for our in-context model (BraInCoRL), the fully trained oracle ("Fully Trained"),, futhin-subbect ridge regression baselines (100, 300), and the FsAverage map across five category- selective regions (faces, places, bodies, words, food).
Faces
Places
Bodics
Words
Food
Mcan
S1
S2
S1
S2
S1
S2
S1
S2
S1
S2
S1
S2
Fully Trained
0.15
0.13
0.16
0.22
0.24
0.20
0.08
0.08
0.13
0.14
0.14
0.15
Ridge-100
0.03
0.02
0.03
0.05
0.07
0.05
0.00
0.01
0.02
0.03
0.02
0.03
Ridge-300
0.07
0.05
0.07
0.10
0.14
0.10
0.03
0.03
0.05
0.05
0.06
0.06
PsAAerage map
0.13
0.06
0.11
0.19
0.09
0.08
0.06
0.03
0.14
0.18
0.08
0.06
BraInCoRLL-100
0.14
0.12
0.15
0.21
0.23
0.18
0.07
0.07
0.11
0.12
0.12
0.14
Table S.6: Voxel-wise explained variance with the DINO backbone for Subjects 3 and 4. We report performance for our in-context model (BraInCoRL), the fully trained oracle ("Fully Trained"), within-subject ridge regression baselines (100, 300), and the FsAverage map across five category- selective regions (faces, places, bodies, words, food).
Faces
Places
Bodies
Words
Food
Mean
$3
S4
S3
S4
S3
S4
S3
S4
$3
S4
S3
S4
Fully Trained
0.13
0.11
0.13
0.12
0.13
0.14
0.06
0.05
0.08
0.09
0.10
0.11
Ridge-100 100
0.02
0.00
0.03
0.01
0.03
0.02
-0.01
-0.02
0.00
-0.02
0.01
0.01
Ridge-300
0.05
0.04
0.06
0.04
0.06
0.07
0.02
0.01
0.03
0.03
0.04
0.04
FsAverage map
0.10
0.03
0.14
0.05
0.11
0.06
0.07
0.03
0.10
0.07
0.09
0.04
BraInCoRI-1100
0.11
0.10
0.13
0.12
0.13
0.14
0.05
0.05
0.06
0.08
0.09
0.10
Table S.7: Voxel-wise explained variance with the DINO backbone for Subjects and 6. We report performance for our in-context model (BraInCoRL), the fully trained reference ("Fully Trained"), within-suubeet ridge regression baselines (100, 300), and the FsAverage map across five category- selective regions (faces, places, bodies, words, food).
Faces
Places
Bodies
Words
Food
Mean
S5
S6
85
S6
SS
S6
S5
S6
S5
S6
S5
S6
Fully Trained
0.19
0.14
0.27
0.11
0.22
0.15
0.13
0.06
0.20
0.06
0.19
0.08
Ridge-100
0.04
0.02
0.05
0.01
0.07
0.03
0.01
-0.01
0.03
-0.01
0.04
0.01
Ridge-300
0.08
0.06
0.13
0.04
0.12
0.07
0.04
0.01
0.08
0.02
0.09
0.03
PsAverage map
0.07
0.05
0.11
0.08
0.06
0.04
0.05
0.06
0.08
0.04
0.07
0.05
BraInCoRI--100
0.18
0.12
0.26
0.10
0.21
0.14
0.10
0.04
0.18
0.04
0.17
0.07
37

---

Table S.8: Voxel-wise explained variance with the DINO backbone for Subjects and 8. We report performance for in-context model (BralnCoRL), the fully trained oracle ("Fully Trained"), within-subject ridge regression baselines (100, 300), and the FsAverage map across five category- selective regions (faces, places, bodies, words, food).
Faces
Places
Bodies
Words
Food
Mean
S7
S7
S8
S7
S7
Fully Trained
0.10
0.06
0.15
0.08
0.18
0.07
0.08
0.02
0.09
0.03
0.12
0.07
Ridge-100
0.02
-0.00
0.02
0.00
0.06
0.00
0.01
-0.01
0.00
-0.00
0.02
-0.00
Ridge-300
0.05
0.02
0.06
0.03
0.10
0.03
0.03
0.00
0.02
0.01
0.05
0.02
FsAverage map
0.12
0.04
0.17
0.04
0.10
0.03
0.09
0.03
0.19
0.02
0.09
0.03
BraInCoRI-100
0.10
0.06
0.14
0.07
0.17
0.07
0.02
0.07
0.04
0.06
Table S.9: Voxel-wise explained variance with the SigLIP backbone for Subjects 1 and 2. We report performance for our in-context model (BraInCoRL), the fully trained oracle ("Fully Trained"), within-subject ridge regression baselines (100, 300), and the FsAverage map across five category- selective regions (faces, places, bodies, words, food).
Faces
Places
Bodics
Words
Food
Mean
S1
S2
S1
S2
S1
82
S1
S2
S1
S2
S1
S2
Fully Trained
0.19
0.17
0.21
0.27
0.30
0.25
0.12
0.11
0.17
0.18
0.19
0.20
Ridge-100
0.10
0.07
0.09
0.14
0.18
0.12
0.03
0.04
0.06
0.07
0.08
0.08
Ridge-300
0.14
0.11
0.14
0,20
0.23
0.17
0.07
0.06
0.11
0.12
0.12
0.13
FsAverage map
0.13
0.06
0.11
0.19
0.09
0.08
0.06
0.03
0.14
0.18
0.08
0.06
BraInCoRI--100
0.17
0.13
0.18
0.24
0.27
0.21
0.09
0.08
0.13
0.14
0.15
0.16
Table S.10: Voxel-wise explained variance with the SigLIP backbone for Subjects 3 and 4. We report performance for our in-context model (BralnCoRL), the fully trained oracle ("Fully Trained"), within-subjeet ridge regression baselines (100, 300), and the FsAverage map across five category-selective regions (faces, places, bodies, words, food).
Faces
Places
Bodies
Words
Food
Mean
$3
S4
S3
S4
S4
S3
S3
S4
Fully Trained
0.17
0.14
0.17
0.17
0.18
0.18
0.10
0.08
0.11
0.13
0.14
0.15
Ridge-100
0.07
0.04
0.08
0.06
0.09
0.07
0.02
0.01
0.03
0.04
0.05
0.05
Ridge-300
0.11
0.08
0.11
0.10
0.12
0.12
0.05
0.03
0.06
0.07
0.08
0.09
FsAverage map
0.10
0.03
0.14
0.05
0.11
0.06
0.07
0.03
0.10
0.07
0.10
0.04
Bra1nCoRL-100
0.12
0.11
0.13
0.14
0.14
0.15
0.05
0.05
0.06
0.09
0.10
0.12
Table S.11: Voxel-wise explained variance with the SigLIP backbone for Subjects 5 and We report performance for our in-context model (BraInCoRL., the fully trained reference ("Fully Trained"), within-subject ridge regression baselines (100, 300), and the FsAverage map across five category-selective regions (faces, places, bodies, words, food).
Faces
Places
Bodics
Words
Food
Mcan
S5
S6
S5
S6
S5
S6
Fully Trained
0.24
0.18
0.33
0.15
0.28
0.19
0.18
0.10
0.26
0.10
0.24
0.12
Ridge-100 100
0.11
0.07
0.16
0.04
0.14
0.09
0.06
0.01
0.11
0.02
0.11
0.04
Ridgo-300
0.17
0.12
0.24
0.08
0.20
0.14
0.10
0.04
0.17
0.05
0.16
0.07
PsAverage map
0.07
0.05
0.11
0.08
0.06
0.04
0.05
0.06
0.08
0.04
0.07
0.05
BraInCoRI--100
0.20
0.14
0.28
0.11
0.23
0.16
0.12
0.05
0.19
0.05
0.19
0.09
38

---

Table S.12: Voxel-wise explained variance with the SigLIP backhone for Subjects 7 and 8. We report performance for our in-context model (BraInCoRL), the fully trained oracle ("Fully Trained"), within-subject ridge regression baselines (100, 300), and the FsAverage map across five category-selective regions (faces, places, bodies, words, food).
Faces
Places
Bodies
Words
Food
Mean
S7
S8
S7
S8
S7
S8
S7
S8
S7
S8
S7
S8
Fully Trained
0.14
0.09
0.15
0.10
0.22
0.09
0.13
0.04
0.13
0.06
0.16
0.09
Ridge-100
0.06
0.02
0.08
0.04
0.11
0.03
0.03
0.00
0.03
0.02
0.06
0.03
Ridge-300
0.10
0.05
0.07
0.15
0.05
0.02
0.06
0.09
0.05
PsAverage map
0.12
0.04
0.17
0.04
0.10
0.03
0.09
0.03
0.19
0.02
0.09
0.03
BraInCoRL-1000
0.11
0.07
0.08
0.19
0.08
0.02
0.09
0.12
0.07
39

---

A.7 Voxelwise explained variance across varying support set sizes for more subjects, backbones and for pretrain-only models
In this section, we investigate how the size of the in-context support set affects voxelwise predictive performance. We evaluate three image-encoding backbones (CLIP, DINO, SigLIP) on eight subjects (S1-S8) by comparing the performance of BraInCoRI. with the pretrain-oni, BraInCoRI. (i.e. same architecture but only pretrained), the within-subject reggessegnnnonn baseline, and the fully trained reference model fit to converge on each subject's 's full 9,000-image training set.
CLIP
DINO
SIGLIP
Subj Explained Variance
0.15
9:35
0:30
0.1
0.30
0.05
0.55
Coli
BranCaR
TraincoaL
Protranned
Prctroined
Pretriined
Midge Basellne
Ridge Baseine
Ridge Baselite
Nuy Faited
Fully Trained
fuly Fained
0.05
0.05
0.05
Explained Variance
0.15
0.25
Subj
0:30
0.10
0.30
0.05
0.05
BraincAAL
BranCoof
BrainCuAL
Fretrained
Pretrained
Fretrained
Ridge Besaline
Ridge Dassl
0.00
Rigg Daneline
Fuly Psined
Fully Trained
Fuly Trined
-3.0
.0.5
0.05
Explained Variance
0.100
0.100
Subj 3
0.075
0.075
0.075
0.000
0.080
0.050
0.025
BraincAAL
0.025
BranCooi
0.025
BrainCuAL
0.000
Protrinedd
0.000
Prctroincd
0.000
Protrined
-0.025
Tuly Traned
Ridg Baseline
-0.025
Ridge Basal
Fully Trained
-0.025
Fully Trained
Ridge Baselise
-0.050
-0.00 050
-0.050
Explained Variance
0.150
0.150
Subj 4
0.075
0.050
0.050
0.025
0.025
BrinCCRR Poetrained
BratnCoo
0.000
Pretratned
0.000
Freirained
.0.025
Fully Tained
Ride Baseline
.0.005
Fully Traindd
Ridge Basal no
.0.02
Ride Baselise Fully Talned
-0 050
-0.050
-0.050
200
800
1000
620
600
1000
600
800
1000
Examples
Exampies
In-Canbeat Examples
Figure S.7: Voxelwise explained variance function of in-context support set size. Voxelwise explained variance is visualized for subjects 1, 2, 3, 4 for each backbone (CLIP, DINO, SigLIP). we plot results for the BraInCoRL, along with the pretrain-only BraInCoRL (i.e. same architecture but only pretrained), the within-subject ridge-regression baseline, and the fully trained reference model using all 9,000 images.
40

---

CLIP
DINO
SIGLIP
Subj Explained Variance
0:0
0.0
0.10
8.05
0.05
0.05
CoAL
BrainCoRL
BrainCoAL
Prutraneed
Pretrained
Prurained
Noge Biseline
Rudge Basslin
Nage Baselise
Petted
rudly Twined
Tuly Telred
-0.05
-0.0
Explained Variance
0.15
0.35
Subj
9.10
0.10
0.10
0.05
BrainCoRL
BrainCoAL
Pretriined
Pretrained
Fretriined
Nddge Beseline
Rilge Basel
Ridge Baselis
Fuly Fained
Fully Trined
Fuly Talned
-2.05
-005
-3.05
Explained Variance
0.325
0.325
0:300
Subj
0.050
0.460
0.050
0.025
Drain
0.025
BrainCalL
0.025
SeatnCoRL
0.000
Prstraired
0.000
Pretrined
0.000
Fratrained
-0.025
Tull Tsited
Rige Broeline
0.025
Ridge Basel
Fidly Thained
-.022
Tuly Trired
Ridge Baselite
0.050
0.050
0.050
Explained Varlance
Subj
0.850
0.025
Brain
0.025
BrainCoRL
0.025
BrainCoAL
0.000
Pretrained
0.000
Pretroined
0.000
Protrained
-0 03.025
Fulyy Vained
Nege Baseline
-0.025
Radge Basel ne
Fully Trained
-0.025
Fully Fained
Nage Baseline
-0 050
-0,050
-0.00 050
1000
1000
of In-Contaxt Examples
Number of In-Contect Examples
Number In-Contaxt Examples
Figure S.8: Voxelwise explained variance function of in-context support set size. Voxelwise explained variance is visualized for subjects 5, 6. 7, 8 for each backbone (CLIP, DINO, SigLIP). we plot results for the BraInCoRL, along with the pretrain-only BraInCoRL (i.e. same architecture but only pretrained), the within-subject ridge-regression baseline, and the fully trained reference model using all 9,000 images.
41

---

A.8
Impact of holding out the test subject's unique images during meta-training evaluated on more backbones
In this section, we further conduct ablations by evaluating BraInCoRL model trained without holding out the test subject's support images ("BraInCoRL no HO") and a BraInCoRL model with only pretraining, on DINO and SigI.I .IP backbones. The result indicates that fine-tuning real neural data enhances performance, and that BraInCoRI. able to generalize effectively entirely unseen images without having encountered them during training.
0.25
0.20
0.15
0.10
0.05
0.00
Ridge
-0.05
-0.10
Subj]
Subj2
Subj5
Subj7
Figure S.9: Distributions of voxelwise explained variance for subjects 1, 2, and 7 using DINO encoding. Results confirm that finetuning with real neural data boosts performance and that BraInCoRL can generalize well to previously unseen images without requiring them during training.
0.25
0.20
0.15
0.10
0.05
-0.05
Sub/1
Subj2
Subj5
Subj7
Figure S.10: Distributions of voxelwise explained variance for subjects 1, 2, 5 and 7 using SigLIP encoding. Results confirm that finetuning with real neural data boosts performance and that BraInCoRL can generalize well to previously unseen images without requiring them during training.
42

---

A.9 Correlation of each backbone's predictions with fully trained activation predictions
Using various image-encoding backbones, we plot how each subject's ss BraInCoRL predicted explained variance correlates with the fully trained model's explained variance (fully trained model refers to the fully trained reference model fit to converge on each subject's full 9,000-image training set), Across every backbone and all subjects, this correlation remains uniformly high.
Fully Trained
0.6
y
y
Subj1
0.4
0.2
0.0
Fully Trained
0.6
y=x
y=x
y =
Subj2
Fully Trained
0.6
=
y
Subj3
0.4
0.2
0.0
Fully Trained
0.6
y=x X
X
y _ X
Subj4
0.4
CLIP
DINO
SIGLIP
Figure S.11: Voxelwise polained-varianee correlation across backbones for subject 1, 2, 3, Each panel shows a scatter the BraInCoRL's 's explained-variance predictions (100 in-context examples) versus the fully trained reference model's explained variance for each voxel. Rows correspond to subjects (1, 2, 3, 4) and columns to image-encoding backbones (CLIP, DINO, SigLIP). The dashed line marks y
43

---

Fully Trained
0.6
y X
Subj5
Fully Trained
0.6
y y=x
y
y X
Subj6
0.4
02
0.0
Fully Trained
0.6
=
y
y = x
Subj7
0.4
02
0.0
Fully Trained
0.0
==x
==x
y=x
Subj8
0.2
0.0
CLIP
DINO
SIGLIP
Figure S.12: Voxelwise explained-variance correlation across backbones for subject 5, 6, 7, 8. Each panel shows a scatter of the BraInCoRL's explained-variance predictions (100 in-context examples) versus the fully trained reference model's explained variance for each voxel. Rows correspond subjects (5, 6. 7, and columns to inage-encoding backbones (CLIP, DINO, SigLIP). The dashed line marks y = T.
44

---

A.10
Voxelwise explained-variance evaluation in BOLD5000 for more subjects and different backbones
In this section, we analyze how varying the number of in-context examples influences voxel-level prediction performance on the BOLD5000 dataset. For subject S2 and S3, we plot the mean Pearson's between
and actual BOLD responses as function of support-set size, comparing our BraInCoRI. model against ridge regression baseline. Results are averaged over five cross- validation folds. Across all three image-encoding backbones (CLIP, DINO, and SIGLIP), BraInCoRI. consistently outperforms ridge regression.
0.20
Pearson r
CS12
0.10
BrainCooL
BrttnCo
BrainCoRL Ridge Baselig ie
Ridge Baselne
Ridge Saseline
0.00
0.20
CSI3 Pearson r
0.15
0.05
20
40
60
80
20
40
60
80
20
40
60
80
# In-Context Samples CLIP
In-Context Samples DINO
# In-Context Samples SIGLIP
Figure S.13: Support-set size versus voxelwise Pearson r in BOLD5000. Fach panel shows the mean voxelwise Pearson correlation between predicted and actual BOLD5000 responses for BraInCoRL and ridge regression, plotted against the number of in-context samples.
45

---

A.11
Dimensional reduction of predicted response function weights on more subjects
In this section, we utilize UMAP visualize the BrainCoRL-preditter voxelwise weights under the CLIP image encoding backbone for subject S1-S8. The cortical maps show color-coded mappings that align well with functionally-deii regions.
Superror
UMAAS
Anterior
UMAP2
UMAAP
RSC
RSC
OPA
OPA
EBA
Food
EEA
Food
reglon
FFFA2
Faces
IPPA
PPA
Subj1
Subj2
Subj3
Left Hemisphere
Subj4
Right Hemisphere
Figure S.14: Dimensional reduction of predicted response weights for subject S1-S4 under CLIP backbone. The cortical maps show color-coded mappings that align well with functionally-defin regions: body and face regions (EBA and FFA/aTL-faces), place regions (RSC/OPA/PPA).. and food regions (in red).
46

---

Superior
LIMAP3
Anterior
Anterto
UMAP2
UMAPT
RSC
RSC
OFA
rogion
Food
PEA
EBA
Food
FTA-1
region
faces
PPA
PPA
faces
Subj5
Subj6
Subj7
Left Hemisphere
Subj8
Right Hemisphere
Figure S.15: Dimensional reduction of predicted response weights for subject S5-S8 under CLIP backbone. The cortical maps show color-coded mappings that align well with functionally-defin regions: body and face regions (EBA and FFA/ATLLffcees), place regions (RSC/OPA/PPA),, and food regions (in red).
47

---

A.12 Predicting cortical responses from natural language prompts on more subjects
In this section, we further predict cortical responses from natural language prompts on subject 2-8. For each semantic category, we convert a natural language prompt into a CLIP text embedding, project it into the image feature space, and use BraInCoRL to predict the corresponding voxel activation map. The predicted activations align closely with known /-statistic of category-selective region, illustrating the potential for zero-shot, language-driven functional mapping of visual cortex.
vmin
vmax
True t-statistics
probing
"A photo torso"
photo of person's 's face"
"A photo bedroom"
photo words"
Figure S.16: Predicting responses of natural language prompts for subject 2. We convert each text prompt corresponding to a semantic category into CLIPP text embedding, project into image-feature space, and predict its cortical activation on subject 2. The resulting activation maps closely match the i-statistics of known category-selective regions, demonstrating the feasibility of language-driven, zero-shot functional mapping of the visual cortex.
vmin
True r-statistics
probing
"A photo of torso"
photo of person's face
"A photo bedroom"
photo of words"
Figure S.17: Predicting responses of natural language prompts for subject 3. We convert each text prompt corresponding to semantic category into CLIP text embedding, project it into image-feature space, and predict its cortical activation on subject The resulting activation maps closely match the i-statistics of known category-selectioe regions, demonstrating the feasibility of language-driven, zero-shot functional mapping the visual cortex.
48

---

vmin
True t-ttatisties
probing
"A photo of torso"
"A photo of person's face
"A photo of bedroom"
"A photo of words"
Figure S.18: Predicting responses natural language prompts for subject 4. convert each text prompt corresponding to semantic category into CL.P text embedding, project it into image-feature space, and predict cortical activation on subject The resulting activation maps closely match the t-statistics of known category-selectioe regions, demonstrating the feasibility of language-driven, zero-shot functional mapping the visual cortex.
vmin
True t-statistics
probing
"A photo of torso"
"A photo of person's face
"A photo of bedroom"
"A photo of words"
Figure S.19: Predicting responses natural language prompts for subject 5. convert each text prompt corresponding to semantic category into CLIP text embedding, project it into image-feature space, and predict its cortical activation on subject The resulting activation maps closely match the /-statistics of known category- selective regions, demonstrating the feasibility of language-driven, zero-shot functional mapping the visual cortex.
49

---

vmin
True t-statistics
probing
"A photo of torso"
"A photo of person's face"
"A photo of bedroom"
"A photo of words"
Figure S.20: Predicting responses natural language prompts for subject 6. convert each text prompt corresponding to semantic category into CLIP text embedding, project it into image-feature space, and predict cortical activation on subject The resulting activation maps closely match the t-statistics of known category-selectioe regions, demonstrating the feasibility of language-driven, zero-shot functional mapping the visual cortex.
vmin
True t-statistics
probing
"A "Aoto of torso"
"A photo of person's face
"A photo of bedroom"
"A photo of words"
Figure S.21: Predicting responses natural language prompts for subject 7. We convert each text prompt corresponding to semantic category into CLIP text embedding, project it into image-feature space, and predict its cortical activation on subject The resulting activation maps closely match the /-statistics of known category- selective regions, demonstrating the feasibility of language-driven, zero-shot functional mapping the visual cortex.
50

---

vmin
True E-statistics
probing
"A photo of torso
"A photo of person's face"
"A photo of bedroom"
photo of words"
Figure S.22: Predicting responses of natural language prompts for subject 8, We convert each text prompt corresponding to semantic category into a CLIP text embedding, project it into mmae--fcaurre space, and predict cortical activation on subject The resulting activation maps closely match the /-statistics of known ditegory-selectivy regions, demonstrating the feasibility of language-driven, zero-shot functional mapping the visual cortex.
51

---

A.13 Voxelwise prompt classification accuracy for more subjects
In this section, we further quantify the semantic specificity of BraInCoRL's voxelwise predictions on subject 3-8. we compute, for each subject and each category ROI, the fraction of voxels whose peak predicted activation corresponded to the semantic category named by the text prompt.
Table S. 13: Voxelwise prompt classification accuracy for subjects 3 and 4. Each cell shows the percentage of voxels in given category selective region (columns) whose peak predicted activation was elicited by a specific semantic prompt (rows, see Appendix) for subject 3 and 4. Using only 100 support images, BraInCoRI. effectivvvyyy localizes category regions with high data efficiency.
Bodies
Faces
Places
Food
Words
S3
S4
S3
S4
S3
S4
S3
S4
S3
S4
Bodies
0.57
0.42
0.23
0.12
0.04
0.03
0.16
0.20
0.12
0.19
Faces
0.29
0.36
0.60
0.66
0.02
0.03
0.11
0.05
0.20
0.12
Places
0.04
0.08
0.02
0.06
0.84
0.82
0.15
0.20
0.08
0.16
Food
0.07
0.09
0.14
0.12
0.09
0.09
0.53
0.51
0.51
0.43
Words
0.02
0.04
0.01
0.04
0.01
0.02
0.05
0.05
0.09
0.10
Table 1.14: Voxelwise prompt classification accuracy for subjects 5 and 6. Each cell shows the percentage of voxels in given category selective region (columns) whose peak predicted activation was elicited by a specific semantic prompt (rows, see Appendix) for subject 5 and 6. Using only 100 support images, BraInCoRL effectively localizes category-sslectie regions with high data efficiency.
Bodies
Faces
Places
Food
Words
S5
S6
S5
S6
S5
S6
S5
S6
S5
S6
Bodies
0.54
0.64
0.17
0.21
0.01
0.08
0.19
0.15
0.27
0.25
Faces
0.29
0.25
0.65
0.63
0.00
0.04
0.03
0.05
0.20
0.15
Places
0.06
0.01
0.05
0.01
0.88
0.65
0.13
0.09
0.10
0.04
Food
0.09
0.07
0.13
0.11
0.10
0.20
0.64
0.66
0.39
0.41
Words
0.02
0.03
0.01
0.04
0.00
0.04
0.01
0.06
0.05
0.15
Table S.15: Voxelwise prompt classification accuracy for subjects 7 and 8. Each cell shows the percentage of voxels in given category selective region (columns) whose peak predicted activation was elicited by a specific semantic prompt (rows, see Appendix) for subject and Using only 100 support images, BraInCoRL effectively localizes category-selectin regions with high data efficiency.
Bodies
Faces
Places
Food
Words
S7
S8
S7
S8
S7
S8
S7
S8
S7
S8
Bodies
0.69
0.57
0.26
0.15
0.01
0.07
0.08
0.20
0.19
0.17
Faces
0.19
0.25
0.59
0.59
0.01
0.04
0.04
0.10
0.22
0.18
Places
0.04
0.05
0.02
0.03
0.89
0.58
0.12
0.06
0.13
0.05
Food
0.06
0.10
0.11
0.20
0.07
0.26
0.68
0.57
0.32
0.47
Words
0.02
0.03
0.03
0.03
0.01
0.04
0.07
0.08
0.14
0.12
52

---

A.14 Additional evaluation of BraInCoRL on NSD dataset
In this section, we provide two more evaluation metrics, namely Pearson and Spearman's 's rank correlation coefficient (Spearman's s p) for NSD dataset on Subject 1. In this case, the BrainCoRL model has not been trained or finetuned on Subject while the Fully Trained model is trained on 9,000 images from this subject.
Subj1
0.35
Pearson
0.30
BrainCoRL
0.20
Ridge Baseline
Fully Trained
0.15
200
800
1000
In-Context Samples
Figure S.23: Voxel-wise Pearson R for BraInCoRL, within-subject ridge regression baseline and fully-trained reference model (NSD dataset, CLIP backbone, Subject 1, higher is better).
Subj1
0.35
Spearman's p
0.30
0.25
0.20
BrainCoRL Ridge Baseline Fully Trained
0.15
0.10
200
400
600
800
1000
In-Context Samples
Figure $.24: Voxel-wise Spearman's p for BraInCoRI., within-subject ridge regression baseline and fully-trained reference model (NSD dataset, CLIP backbone, Subject 1, higher is better).
53

---

A.15 Additional evaluation of BraInCoRL on BOLD5000 dataset
In this section, we provide two more evaluation metries, namely explained variance and Spearman's rank correlation coefficient (Spearman's 's p) for BOLD5000 dataset on Subject CSI1.
CSI1
.050
Expl. Variance
.025
.000
BraInCoRL
Ridge Baseline
20
40
60
80
In-Context Samples
Figure S.25: Voxel-wise explained variance for BraInCoRLand within-subject ridge regression baseline (BOLD5000 dataset, CLIP backbone, Subject CSIL, higher is better).
CSI1
0.20
Spearman's p
0.15
0.10
BrainCoRL
0.05
Ridge Baseline
20
40
60
80
# In-Context Samples
Figure S.26: Voxel-wise Spearman's 's for BrainCoRLand within-subject ridge regression baseline (BOLD5000 dataset, CLIP backbone, Subject CSI1, higher better).
54

---

A.16 Evaluation of each training stage's contribution
In this section, we present the voxelwise explained variance for BraanCoRLtrainedd until different training stages. The results show that the progression from synthetic foundation + context flexibility biological adaptation ensures that each fundamental challenge, namely response function coverage, variable context handling, and biological realism, is systematically addressed in the optimal order.
Subj1
0.18
0.16
Expl. Variance
0.10
0.08
Stage
0.06
Pretrain Stage
Fine-tuned
0.04
Ridge Baseline
Ground Truth
200
400
600
800
1000
In-Context Samples
Figure S.27: Voxelwise explained variance for BraInCoRL CLIP backhone for NSD Subject on different training stages, compared with ridge baseline and fully trained reference model (higher better).
Subj2
0.175
Expl. Variance
0.150
0.125
Pretrain Stage
0.075
Pretrain Stage
0.050
Fine-tuned
Baseline
0.025
Ground Truth
200
400
600
800
1000
In-Context Samples
Figure S.28: Voxelwise explained variance for BralnCoRLwith CLIP backbone for NSD Subject on different training stages, compared with ridge baseline and fully trained reference model (higher better).
55

---

A.17 Performance of BraInCoRL conditioned on the full 9000-image set
In this section, we evaluate the performance of BraInCoRL with 9,000 in-context samples, for the four NSD subjects (S1, S2, S5, S7) focues by our main paper. The performance difference is less than 1% across all subjects when compared subject's entire 9.000-image training set using gradient descent over multiple epochs). This means BraInCoRI. achieves 94-99% of the fully trained model's performance.
fully trained model (which is fit converge on each
Table S.16: Voxel-wise explained variance of BraInCoRL with the CLIP backhone compared with the fully trained reference model. The difference variance explained is less than %.
Subject
Method
S1
S2
S5
S7
Fully Trained
0.1765
0.1882
0.2310
0.1554
BrainCoRL
0.1667
0.1817
0.2225
0.1541
56

---

A.18 Evaluation on the choice of loss function during training
In this section, we conducted an ablation study on the choice of different loss functions during the finetuning stage optimization and evaluated the model performance.
Our experimental results show that MSE and LI losses achieve similar performance across all context sizes, with minimal differences. This suggests that both metrics are equally effective for capturing voxelwise neural response patterns.
In addition, the hybrid loss of 0.5 x MSE loss + 0.5 x (1 cosine similarity) underperforms by approximately 2-4% compared to MSE/LI. We argue this is because although cosine similarity captures directional relationships between predicted and true responses, this additional constraint limit the model's ability to accurately predict response magnitudes.
Subj1
0.20
Expl. Variance
0.15
0.10
0.05
MSE + CosSim
Fully
0.00
250
500
750
1000
In-Context Samples
Figure S.29: Voxel-wite explained variance of different training losses with the CLIP backbone for Subject (higher is better).
57

---

A.19 Ablation on logit scaling
In this section, we provide an additional ablation study evaluate the effect of the logit-scaling. where we report the Voxel-wise explained variance of our BrainCoRL model and the model with the exact same structure but without logit scaling. is shown that the logit scaling significantly boosts the model performance and generizability of various in-context support set sizes.
Subj1
0.20
Expl. Variance
0.15
BrainCoRL No Logit Scale
0.10
Fully Trained
0.05
0.00
250
500
750
1000
# In-Context Samples
Figure S.30: Voxel-wise explained variance for BrainCoRL with CLIP backbone for Subj 1, compared to the same model architecture but without logit scaling (higher is better).
58

---

Appendix References
[1] Jianlin Su. Analyzing the scale operation of attention from the perspective of entropy invariance, Dec 2021. URL https: / /wweueeeeeeee///e//ee/ fm/ archives/880 /8823.
David Chiang and Peter Cholak. Overcoming theoretical limitation of self-attention. arXiv preprint arXiv:2202. /2/72, 2022.
[3] Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, al. Qwen technical report. arXiv preprint arXiv:2309.16609, 2023.
[4] Meta AL The llama herd: The beginning of a new era of natively mul- timodal ai innovation. 4(7):2025, 2025. URL https://aas wwt.a.tt.aaaa....eeeeeee com/blog/ checkedon.
1lama-4-mae de-mmtmmmdallinteeeeee
59


Reviews:
[{"review_id": "z0g7A4vTsc", "reviewer": "Reviewer_H8UU", "summary": "This paper addresses an important problem: how to construct cross-subject, few-shot fMRI encoding models. From my understanding, the paper appears to use a small set of image- (single) voxel pairs as a support set, and based on this, trains a model to output the parameters of voxel-wise encoding model. However, I did not fully grasp some details of the proposed method. The paper evaluates its approach on datasets such as NSD, and the experimental results show that the proposed method can effectively encode fMRI signals.", "questions": "For the methodology section, I have the following unclear points:\n+ In line 131, the context tokens c is indexed by i, which seems to imply that its range is from 1 to p, however, in line 132, its range is from 1 to k. it seems inconsistent?\n+ The authors use beta to represent the voxel's BOLD signal (such as in line 129), but in the context tokens c, they directly use the \"voxel\" v. Does this v here refer to a vector similar to a positional encoding?\n+ The authors employ the technique of logit scaling in Section 3.3; however, I am not familiar with it. Could the authors provide a clearer explanation to clarify its underlying principle?\n\nFor the experimental section, I have the following questions:\n+ What is explained variance? Why is only this metric used in the evaluation on the NSD dataset, while the Pearson correlation coefficient is not assessed?\n+ Are there additional metrics to directly assess the accuracy of fMRI encoding?", "limitations": "None", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n+ The paper addresses an important research problem and achieves promising results.\n+ The paper provides a relatively comprehensive review of related work.\n+ The meta-learning method proposed in the paper is thought-provoking and offers valuable insights.\n\nWeaknesses (Details refer to Questions):\n+ In my opinion, the presentation of the method should be more detailed to improve clarity and comprehensibility.", "was_revised": true, "final_justification": "After carefully reading the authors' response, all of my questions have been fully addressed. This paper is highly insightful for the research field, and therefore I have raised my rating.", "rebuttal": "We appreciate your very thoughtful and helpful suggestions. Below are our responses. We look forward to further discussion, and welcome additional comments.\n\n> **Q1) Notation regarding token indexing**\n\nThe context tokens in `line 129` and `line 132` should be indexed by the same variable and extend to the same count. **This is a typo that we have updated in our upcoming revision.** In both cases, the number of context tokens during inference represents the number of stimuli that a subject has seen -- which are then made available to the model for in-context learning.\n\n**We will update this in the revised paper.**\n\n> **Q2) Notation regarding voxel signal**\n\nWe agree that the clarity here could be improved. Per your suggestion, we have unified the notation around voxel $v$ and beta value $\\beta$. Our original intention was to use $\\beta$ since the voxel's response to a given image in fMRI viewing is typically derived as the beta coefficients of a GLM regression. We intend the two variables to represent the same value, which is a scalar representing a voxel's response to a given stimulus. \n\n**This has been changed in our revision.**\n\n> **Q3) Principles underlying logit scaling**\n\nThe motivating factor underlying logit scaling is our desire to have our in-context learned encoder perform well *regardless of the number of stimuli given to the model*, and effectively generalize to context sizes beyond those seen during training. For example, while we may only train with between $30$ and $500$ images, a third-party experimenter may want to use fewer than $30$ images or more than $500$ images to condition the model. Across all cases, we want the model to succeed.\n\nThis logit scaling method was first proposed in [1, 2], and later adapted in Qwen LLM (logn-scaling) [3] & Llama 4 LLM (temperature scaling) [4]. We will briefly summarize the high-level math, which we take from [1] with our commentary:\n\nLet the attention value in self-attention for a particular query token $i$ to value token $j$ to be\n\n$$a\\_{i,j} = \\frac{e^{\\lambda \\mathbf{q}\\_i \\cdot \\mathbf{k}\\_j}}{\\sum\\_{j=1}^n e^{\\lambda \\mathbf{q}\\_i \\cdot \\mathbf{k}\\_j}}$$\n\nThen the entropy is defined as $$\\mathcal{H}\\_i = -\\sum\\_{j=1}^n a\\_{i,j} \\log a\\_{i,j}$$\n\nSubsituting the expression for $a\\_{i,j}$ we have the entropy as\n\n$$\\mathcal{H}\\_i = -\\sum\\_{j=1}^n a\\_{i,j} \\log \\left(\\frac{e^{\\lambda \\mathbf{q}\\_i \\cdot \\mathbf{k}\\_j}}{\\sum\\_{j=1}^n e^{\\lambda \\mathbf{q}\\_i \\cdot \\mathbf{k}\\_j}}\\right)$$\nSince $\\sum\\_{j=1}^n a\\_{i,j} = 1$, \n\nwe can express the formula as \n\n$$\\mathcal{H}\\_i = \\log \\sum\\_{j=1}^n e^{\\lambda \\mathbf{q}\\_i\\cdot \\mathbf{k}\\_j} - \\frac{\\sum\\limits\\_{j=1}^n e^{\\lambda \\mathbf{q}\\_i\\cdot \\mathbf{k}\\_j}(\\lambda \\mathbf{q}\\_i\\cdot \\mathbf{k}\\_j)}{\\sum\\limits\\_{j=1}^n e^{\\lambda \\mathbf{q}\\_i\\cdot \\mathbf{k}\\_j}}$$\n\nIf the first term is expressed as an expectation over $j$, we have\n\n$$\\sum\\_{j=1}^n e^{\\lambda \\mathbf{q}\\_i\\cdot \\mathbf{k}\\_j} = n\\times \\frac{1}{n}\\sum\\_{j=1}^n e^{\\lambda \\mathbf{q}\\_i\\cdot \\mathbf{k}\\_j}\\approx n\\mathbb{E}\\_j[e^{\\lambda \\mathbf{q}\\_i\\cdot \\mathbf{k}\\_j}]$$\n\nWhich leads to the following approximation of entropy\n\n$$\\mathcal{H}\\_i \\approx \\log n + \\log \\mathbb{E}\\_j[e^{\\lambda \\mathbf{q}\\_i\\cdot \\mathbf{k}\\_j}] - \\frac{\\lambda\\mathbb{E}\\_j[e^{\\lambda \\mathbf{q}\\_i\\cdot \\mathbf{k}\\_j}(\\mathbf{q}\\_i\\cdot \\mathbf{k}\\_j)]}{\\mathbb{E}\\_j[e^{\\lambda \\mathbf{q}\\_i\\cdot \\mathbf{k}\\_j}]}$$\n\nIf the vectors are assumed to be the output of a layernorm layer with length $\\sqrt{d}$, the expectation can be converted to one over the angles between vectors\n\n$$\\mathcal{H}\\_i \\approx \\log n + \\log \\mathbb{E}\\_{\\theta}[e^{\\lambda d \\cos\\theta}] - \\frac{\\lambda d\\mathbb{E}\\_{\\theta}[e^{\\lambda d \\cos\\theta}\\cos\\theta]}{\\mathbb{E}\\_{\\theta}[e^{\\lambda d \\cos\\theta}]}$$\n\nSince most randomly distributed vectors in higher dimensions are orthogonal, we derive a term which can roughly be expressed as\n\n$$\\mathcal{H}\\_i \\approx \\log n + C$$\nwhere $C$ does not depend on the number of tokens $n$. \n\nThis leads to an approximate scaling factor for the logits of $\\log n$ to keep the entropy invariant to context length.\n\nTherefore, we change the standard formulation of attention values by applying a scaling factor of $\\log n$ to the $QK^T$ term, as shown in `Equation (4)` of our paper.\n\nNote that in the above explanation we adopt the notation from [1].\n\n**We have provided a more detailed discussion of the use of logit scaling in the revised version of our paper.**\n\n> **Q4) Why explained variance?**\n\nExplained Variance is a performance measuring metric that quantifies the proportion of the total variance in the fMRI signal that can be predicted by our BraInCoRL encoding model. \n\nWe follow prior work that utilizes the NSD fMRI dataset, which primarily evaluate ***Explained Variance*** as the metric of encoding model performance [5, 6, 7] -- note all three of these papers have one NSD author as co-author.\n\nWe agree that Pearson $R$ is also a possible metric we can use to evaluate encoder performance. In a sense, Pearson $R$ is ***more forgiving metric***. Pearson $R$ is affected neither by scaling, nor by constant bias, while explained variance is only robust to bias. The use of Pearson $R$ reflects an implicit preference for the model to capture the right trend in predicted responses while caring less about capturing the exact value.\n\nNote that BOLD5000 contains distribution shifts for several reasons - for example, it incorporates COCO, ImageNet, and SUN images. With NSD, we explicitly controlled the training and evaluation responses distributions to be the same, making explained variance appropriate as a strong criterion. In contrast, with BOLD5000, even though every voxel was still z-scored, there are still many factors that are different compared to NSD (MRI scanner quality, more variable task). After discussion with BOLD5000 authors, we were advised to evaluate Pearson $R$ for BOLD5000, rather than the more stringent explained variance (see also the GLMSingle paper for a direct comparison of the datasets).\n\n> **Q5) Other metrics for NSD**\n\nHere we evaluate Pearson $R$, and Spearman's rank correlation coefficient (Spearman's $\\rho$) for NSD dataset. Note that in this case, the BraInCoRL model has not been trained or finetuned on `Subject 1`, while the `Fully Trained` model is trained on $9000$ images from this subject. Our model can approach within 10% of the `Fully Trained` model using just $300$ images when evaluated using Pearson $R$.\n\n\n* Voxel-wise Pearson R for BraInCoRL, within-subject ridge regression baseline and fully-trained reference model (NSD dataset, CLIP backbone, Subject 1, higher is better). \n\n|# In-Context Samples|30|70|100|300|512|700|1000|\n|-|-|-|-|-|-|-|-|\n|Fully Trained|0.3934|0.3934|0.3934|0.3934|0.3934|0.3934|0.3934|\n|BraInCoRL|0.2899|0.3289|0.3411|0.3679|0.3757|0.3790|0.3811|\n|Ridge Baseline|0.1776|0.2287|0.2547|0.3059|0.3282|0.3384|0.3506|\n\n* Voxel-wise Spearman's $\\rho$ for BraInCoRL, within-subject ridge regression baseline and fully-trained reference model (NSD dataset, CLIP backbone, Subject 1, higher is better).\n\n|# In-Context Samples|30|70|100|300|512|700|1000|\n|-|-|-|-|-|-|-|-|\n|Fully Trained|0.3736|0.3736|0.3736|0.3736|0.3736|0.3736|0.3736|\n|BraInCoRL|0.2721|0.3088|0.3215|0.3489|0.3570|0.3604|0.3636|\n|Ridge Baseline|0.1304|0.1969|0.2218|0.2849|0.3070|0.3173|0.3289|\n\n> **Q6) Other metrics for BOLD5000**\n\nHere we evaluate explained variance and Spearman's rank correlation (Spearman's $\\rho$) for BOLD5000. Our BraInCoRL model was trained on NSD, while the Ridge Baseline was fit on BOLD5000. We achieve significantly better performance.\n\n* Voxel-wise EV for BraInCoRL and within-subject ridge regression baseline (BOLD5000 dataset, CLIP backbone, Subject CSI1, higher is better).\n\n|# In-Context Samples|10|20|30|40|50|60|70|80|\n|-|-|-|-|-|-|-|-|-|\n|BraInCoRL|0.0114|0.0155|0.0259|0.0355|0.0421|0.0454|0.0504|0.0531|\n|Ridge Baseline|0.0041|0.0079|0.0057|0.0035|0.0093|0.0155|0.0244|0.0266|\n\n* Voxel-wise Spearman's $\\rho$ for BraInCoRL and within-subject ridge regression baseline (BOLD5000 dataset, CLIP backbone, Subject CSI1, higher is better).\n\n|# In-Context Samples|10|20|30|40|50|60|70|80|\n|-|-|-|-|-|-|-|-|-|\n|BraInCoRL|0.1130|0.1382|0.1561|0.1694|0.1769|0.1846|0.1878|0.1905|\n|Ridge Baseline|0.0790|0.0916|0.1016|0.1192|0.1317|0.1411|0.1480|0.1508|\n\n>**Conclusion**\n\nThank you for the detailed and thoughtful feedback. Following your suggestions, we have clarified our motivation with utilizing logit scaling in `section 3.3`, we have provided reasons why explained variance was used for NSD, and we have run additional experiment to evaluate different regression metrics on NSD and BOLD5000. **We will update our revision to incorporate these changes.**\n\nWe hope our answers address your questions, and look forward to any additional discussion.\n\n\n[1] Analyzing the scale operation of attention from the perspective of entropy invariance (2021). (see line 564 of main paper).\n\n[2] Overcoming a Theoretical Limitation of Self-Attention (2022).\n\n[3] Qwen technical report (2023).\n\n[4] The Llama 4 herd: The beginning of a new era of natively multimodal ai innovation (2025).\n\n[5] Non-neural factors influencing BOLD response magnitudes within individual subjects (2022).\n\n[6] A 7T fMRI dataset of synthetic images for out-of-distribution modeling of vision (2025).\n\n[7] Better models of human high-level visual cortex emerge from natural language supervision with a large and diverse dataset (2023)."}, {"review_id": "YsIDfx8r05", "reviewer": "Reviewer_yVtB", "summary": "This paper introduces BraInCoRL, a novel transformer-based model for brain encoding that performs meta-learning and in-context learning to predict voxel-level responses in the human higher visual cortex to visual stimuli. Unlike traditional brain encoders that require subject-specific large-scale data, BraInCoRL is trained across subjects and generalizes to novel subjects and datasets using only a small number of in-context examples (i.e., image–response pairs).", "questions": "Complexity and Scalability of the Training Pipeline\n\nThe proposed three-stage training strategy—synthetic pretraining, context extension, and real-data finetuning—is well-motivated and effective. However, it introduces significant complexity and resource requirements. In practical settings, especially those with limited access to large-scale fMRI datasets or compute, this may hinder broader adoption. Additionally, the contribution of each stage to the final model performance is not thoroughly disentangled.\n\nSuggestion for Authors\nIt would strengthen the paper to include a quantitative evaluation of each stage’s contribution to performance. For instance, how much does synthetic pretraining alone contribute to generalization? How critical is the context extension step? Ablations isolating each component, or alternative streamlined training setups, would help clarify the necessity of the full pipeline and guide future, potentially more efficient implementations.\n\nOverall comments:\n\nThis paper presents a compelling advance in the intersection of computational neuroscience and meta-learning. The proposed BraInCoRL model is technically innovative, empirically strong, and provides new insights into how generalizable brain encoding models can be built.", "limitations": "Yes.", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strong Generalization Across Subjects and Datasets\nBraInCoRL shows consistent improvements across novel subjects and even across datasets collected with different scanners and protocols (e.g., NSD to BOLD5000). It operates effectively with just a few in-context images, outperforming ridge regression baselines and closely matching fully trained models using 9,000 images.\n\nModular and Versatile Model Design\nThe model is constructed modularly with a frozen CLIP feature extractor and a higher visual cortex transformer that can generate hyperweights for voxel-wise encoding. This enables efficient processing of both visual and language inputs, facilitating downstream tasks such as natural language–driven cortical querying.\n\nInsightful Analyses on Model Behavior\nThe paper presents compelling insights into the behavior of the model:\nIt learns to attend to semantically relevant images when predicting voxel responses in category-selective regions.\n\nSurprisingly, it performs better when in-context support sets are randomly sampled rather than overly specific, suggesting that semantic diversity enhances generalization—a notable finding for the design of few-shot neural systems.", "was_revised": true, "final_justification": "The rebuttal addressed my questions.", "rebuttal": "We appreciate your excellent suggestions! We will incorporate all of your feedback into our paper.\n\n> **Q1) Training cost and complexity**\n\nWe agree with you that the training stage of our model is more costly than traditional fMRI encoders. However this cost gains us significant generalization ability.\n\nOur current model already achieves strong generalization performance on fMRI data that was collected using a different scanner, voxel size, experiment protocol, and human subjects -- **without requiring any finetuning or training on these smaller datasets**. We also achieve significant data-efficiency gains, demonstrating better performance in the low data regime compared to traditional baselines.\n\nDue to this, our **training & inference code and weights will be made available under a permissive open source license**. Future researchers will be able to adopt our model for their datasets without training from scratch (and so will not be limited to their small datasets), and benefit from the models that we have already trained.\n\n\n> **Q2) Contribution of each stage**\n\nThe three-stage training is essential, with each stage addressing a distinct challenge. \n\n* **1st stage: Synthetic Pretraining** enables comprehensive coverage of the visual response space. It provides the model with an initial and broad feature representation by exposing it to a large volume of synthetically generated data.\n* **2nd stage: Context Extension** trains the model on variable-length contexts, enabling robust length generalization, and allows for test-time context size to be outside of the range seen during training. \n* **3rd stage: Real-Data Finetuning** adapts the model to the statistical properties of real fMRI signals, in order to bridge the gap between synthetic and biological domains. Note that in the main paper, for evaluation fairness we avoid training on the same subject that we evaluate on.\n\nThis progression from synthetic foundation → context flexibility → biological adaptation ensures that each fundamental challenge—response function coverage, variable context handling, and biological realism—is systematically addressed in the optimal order.\n\nDue to OpenReview limitations, here we present results only for the first two subjects of the NSD dataset; however, the observed trends and performance are consistent across all other subjects.\n\n* Voxel-wise explained variance for BraInCoRL with CLIP backbone for NSD Subject 1 on different training stages, compared with ridge baseline and fully trained reference model (higher is better).\n\n| # In-Context Samples | 30  | 70  | 100  | 300  | 512  | 700  | 1000  |\n|--------|-----|-----|-----|-----|-----|-----|-----|\n| Fully Trained (9000 images) | 0.1765 | 0.1765 | 0.1765 | 0.1765 | 0.1765 | 0.1765 | 0.1765 |\n| Synthetic Pretraining (stage 1) | 0.0508 | 0.0886 | 0.1034 | 0.1183 | 0.1177 | 0.1153 | 0.1086 |\n| Context Extension (stage 2) | 0.0461  | 0.0906 | 0.1054 | 0.1314 | 0.1381 | 0.1396 | 0.1415 |\n| **Real-Data Finetuning (stage 3)** | **0.0991**  | **0.1283** | **0.1374** | **0.1571** | **0.1628** | **0.1650** | **0.1662** |\n| Ridge Baseline | 0.0227 | 0.0548 | 0.0697 | 0.1081 | 0.1237 | 0.1311 | 0.1402 |\n\n* Voxel-wise explained variance for BraInCoRL with CLIP backbone for NSD Subject 2 on different training stages, compared with ridge baseline and fully trained reference model (higher is better).\n\n| # In-Context Samples | 30  | 70  | 100  | 300  | 512  | 700  | 1000  |\n|--------|-----|-----|-----|-----|-----|-----|-----|\n| Fully Trained (9000 images) | 0.1882 | 0.1882 | 0.1882 | 0.1882 | 0.1882 | 0.1882 | 0.1882 |\n| Synthetic Pretraining (stage 1) | 0.0579 | 0.0968 | 0.1121 | 0.1354 | 0.1338 | 0.1317 | 0.1257 |\n| Context Extension (stage 2) | 0.0632 | 0.1038 | 0.1157 | 0.1402 | 0.1467 | 0.1502 | 0.1526 |\n| **Real-Data Finetuning (stage 3)** | **0.1128**  | **0.1412** | **0.1516** | **0.1720** | **0.1780** | **0.1796** | **0.1813** |\n| Ridge Baseline | 0.0262 | 0.0624 | 0.0787 | 0.1211 | 0.1358 | 0.1453 | 0.1538 |\n\nIn stage 1, we find that training with synthetic data without context extension limits the performance at higher context sizes. After stage 2, the model gain the ability to consistently improve performance as context size increases. In stage 3, after being exposed to real neural data, our model further improves in performance.\n\n> **Conclusion**\n\nWe thank you for your comments, and we hope that this clarifies our results! We have further clarified the contributions of each training stage, and will update the paper to reflect your suggestions."}, {"review_id": "6fdvQdtQjc", "reviewer": "Reviewer_BDTo", "summary": "This work proposes a meta-learning approach to modeling fMRI recordings in response to visual stimuli. The method allows for better generalization between subjects and even recording conditions, taking a step beyond subject-specific encoding models. A transformer is trained to predict linear-readout weights given a relatively small set of in-context image-feature, voxel-activation pairs. The model is trained on the NSD dataset and tested on held-out subjects, as well as the BOLD5000 dataset. Results show that the proposed method explains more variance than direct optimization using Ridge regression across a variety of in-context training set sizes. Further, the manuscript presents clusterings of predicted predicted readout weights in accordance with known category-selective brain regions. An analysis of the images with highest attention scores in the last transformer layer reveals that images tend to stem from the category preferred by the respective brain region.", "questions": "1. A brief overview of the model architecture would be good to include in the main text.\n\n2. In l.131 should it be [x_i;\\beta_i]? Or what does v_i denote exactly?\n\n3. Equation (4) came very out of the blue for me – perhaps this could be explained more thoroughly\n\n4. Is it possible to use the full 9,000 images as in-context samples, thereby actually reaching the performance of the fully trained model?\n\n5. The results reported on page 6 are somewhat unclear to me. It says '...we evaluated BrainCoRL with each subject's 9000 unique images as in-context support set...'. Then in the next sentence '...using just 100 test(?) images', In the table caption, it says '...uses just 100 in-context images...'. Perhaps this could be clarified a little.\n\n6. In Fig.4 for BOLD5000, why are so few samples used compared to NSD?", "limitations": "yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "**Strengths**\n\n1. The idea of applying meta-learning to fMRI data is interesting and as noted in the manuscript, it is a promising approach towards more subject-general models.\n\n2. The results clearly show data-efficiency and transfer abilities of the proposed model. This has strong relevance for the field.\n\n3. Experiments are quite rigorous, testing multiple models and datasets.\n\n4. Cluster analyses and semantic characterizations of the predicted voxel weights show clear correspondences to established category-selective regions, showing that the model does learn meaningful features.\n\n5. The manuscript is clear and easy to follow for the most part.\n\n**Weaknesses**\n\n1. Fig. 3b) shows that the performance difference between the pretrained model and BrainCoRL is non-zero, but relatively small. A lot of the advantage over Ridge regression is seemingly not attributable to explicit leveraging of the structure of neural data.\n\n2. The variance explained by all models is relatively low, which means the model is competitive, but at a low overall level. We observe that it is able to capture very rough trends like category selectivity, but it is difficult to gauge whether learning more fine-grained features driving single voxels is possible with this method. Since a voxel's tuning properties are inferred by studying the tuning properties of other voxels, I could imagine a smoothing effect where voxels with very distinct tuning properties are not modeled accurately. Therefore, while the feature analyses in Fig. 6-8 are interesting, they are not fully convincing for me. This is a drawback but not a strong criticism for this work – I acknowledge that this can be left for future efforts.", "was_revised": true, "final_justification": "The thorough discussion of the questions I've raised has reinforced my position that this manuscript is suitable for publication.", "rebuttal": "We appreciate the concrete feedback and comments! We address your suggestions below.\n\n> **Q1) Performance gain from real neural data**\n\nThis is a good question. To clarify, `Figure 3b` evaluates performance at a fixed in-context support size of ***100 images on a held out subject***. To further evaluate the full impact of real-data finetuning, we visualize the trends as shown in `Section A.6` on page 30 of the Supplement. We find that finetuning yields substantial performance gains -- over 20% improvement in variance explained across support set sizes compared to the pretrained model. This effect becomes especially pronounced at larger context sizes (>500), where **the synthetic pretrained model often falls below the Ridge regression baseline**, while the **fully trained BraInCoRL model continues to improve**.\n\nBeyond raw performance, we want to highlight how BraInCoRL leverages the structure of neural data. During Stage 3 (real-data finetuning), the model meta-learns across a distribution of voxel-level tasks -- each defined by a voxel’s stimulus-response mapping in a specific subject. BraInCoRL uses a transformer-based hypernetwork conditioned on in-context stimulus-response pairs. This design allows the model to capture shared representational structure across subjects and voxels, and adapt to novel subjects and voxels through in-context learning, without test-time training.\n\n> **Q2) Finer-grained selectivity**\n\nWe agree that current models (our model, as well as those from other researchers) based on linear projections of features from a frozen foundation model vision transformer, impose strong constraints on the final fMRI encoding model. This may make the learning of fine-grained features that drive voxel activity difficult. We would like to clarify that our reported explained variance is not normalized to the effective noise ceiling -- and the explained variance of our models are very competitive with recent state-of-the-art work [1] that utilize a vision transformer as the image backbone.\n\nIn the future, a promising direction for fMRI encoders may be constructing non-linear mappings between backbone features and the brain, or those that can capture a distribution of activations based on stochastic generative models. We anticipate that a primary driver of this type of work will be the availability of higher quality and more diverse fMRI datasets.\n\n> **Q3) Model architecture**\n\nThe architecture of our model is composed of:\n\n(1) An **input projection** MLP layer is applied to each token individually, which maps the stimulus semantics and voxel activation into an embedding. \n\n(2) 20 **Transformer encoder** layers that integrate information across all in-context examples and computes trends for voxel activation across multiple stimuli. \n\n(3) The `[CLS]` token from the final layer goes through an MLP to output a hyperweight which is used to parameterize the final encoder. \n\nOur code is provided on `line 22` of the main paper. We will update the paper revision to include additional model details.\n\n> **Q4) Typo clarification**\n\nIn line 131, we correct the context token $c_i$ to be $=[x_i; \\beta_i]$, so $\\beta$ will be consistent in the previous BOLD signal representation in line 129.\n\n> **Q5) Logit scaling equation clarification**\n\nHere we clarify the use of logit-scaling in `Equation (4)` for our model.\n\nLogit scaling is motivated by the need for robust generalization across varying context sizes. Specifically, it helps the in-context encoder maintain stable performance regardless of the number of conditioning stimuli, including those beyond the training range. Although we train with between $30$ and $500$ images, practical applications may involve fewer than $30$ or more than $500$. Logit scaling ensures the model remains functional and accurate even in these extrapolated settings.\n\nThis logit scaling approach was originally proposed in [2, 3], and later adopted by Qwen LLM (logn-scaling) [4] & Llama 4 LLM (temperature scaling) [5]. Below, we provide a brief summary of the high-level mathematical rationale based on [2], along with our own commentary:\n\nThe self-attention value for query token $i$ to value token $j$ is \n\n$$a\\_{i,j} = \\frac{e^{\\lambda \\mathbf{q}\\_i \\cdot \\mathbf{k}\\_j}}{\\sum\\_{j=1}^n e^{\\lambda \\mathbf{q}\\_i \\cdot \\mathbf{k}\\_j}}$$\n\nThen we define the entropy of query token $i$ as $$\\mathcal{H}\\_i = -\\sum\\_{j=1}^n a\\_{i,j} \\log a\\_{i,j}$$\n\nIf we substitute the expression for $a\\_{i,j}$, we will have\n\n$$\\mathcal{H}\\_i = -\\sum\\_{j=1}^n a\\_{i,j} \\log \\left(\\frac{e^{\\lambda \\mathbf{q}\\_i \\cdot \\mathbf{k}\\_j}}{\\sum\\_{j=1}^n e^{\\lambda \\mathbf{q}\\_i \\cdot \\mathbf{k}\\_j}}\\right)$$\nSince $\\sum\\_{j=1}^n a\\_{i,j} = 1$, \n\nwe can express the entropy as\n\n$$\\mathcal{H}\\_i = \\log \\sum\\_{j=1}^n e^{\\lambda \\mathbf{q}\\_i\\cdot \\mathbf{k}\\_j} - \\frac{\\sum\\limits\\_{j=1}^n e^{\\lambda \\mathbf{q}\\_i\\cdot \\mathbf{k}\\_j}(\\lambda \\mathbf{q}\\_i\\cdot \\mathbf{k}\\_j)}{\\sum\\limits\\_{j=1}^n e^{\\lambda \\mathbf{q}\\_i\\cdot \\mathbf{k}\\_j}}$$\n\nIf the first term is expressed as an expectation over $j$, we have\n\n$$\\sum\\_{j=1}^n e^{\\lambda \\mathbf{q}\\_i\\cdot \\mathbf{k}\\_j} = n\\times \\frac{1}{n}\\sum\\_{j=1}^n e^{\\lambda \\mathbf{q}\\_i\\cdot \\mathbf{k}\\_j}\\approx n\\mathbb{E}\\_j[e^{\\lambda \\mathbf{q}\\_i\\cdot \\mathbf{k}\\_j}]$$\n\nThen we can derive the following approximation of entropy\n$$\\mathcal{H}\\_i \\approx \\log n + \\log \\mathbb{E}\\_j[e^{\\lambda \\mathbf{q}\\_i\\cdot \\mathbf{k}\\_j}] - \\frac{\\lambda\\mathbb{E}\\_j[e^{\\lambda \\mathbf{q}\\_i\\cdot \\mathbf{k}\\_j}(\\mathbf{q}\\_i\\cdot \\mathbf{k}\\_j)]}{\\mathbb{E}\\_j[e^{\\lambda \\mathbf{q}\\_i\\cdot \\mathbf{k}\\_j}]}$$\n\nAssuming that the vectors are the output of a layernorm layer with length $\\sqrt{d}$, the expectation can be converted to one over the angles between vectors\n\n$$\\mathcal{H}\\_i \\approx \\log n + \\log \\mathbb{E}\\_{\\theta}[e^{\\lambda d \\cos\\theta}] - \\frac{\\lambda d\\,\\mathbb{E}\\_{\\theta}[e^{\\lambda d \\cos\\theta}\\cos\\theta]}{\\mathbb{E}\\_{\\theta}[e^{\\lambda d \\cos\\theta}]}$$\n\nSince randomly distributed vectors in high-dimensional spaces are mostly nearly orthogonal, we derive a term that can be approximately expressed as\n\n$$\\mathcal{H}\\_i \\approx \\log n + C$$\nwhere $C$ does not depend on the number of tokens $n$. \n\nThis yields an approximate scaling factor for the logits of $\\log n$, which helps maintain the entropy of the attention distribution across different context lengths.\n\nAs a result, we apply logit scaling with a factor of $\\log n$ to the $QK^T$ term in the attention mechanism, changing the standard attention value formulation.\n\nNote that in the explanation we adopt the notation from [2].\n\n**We have provided a more detailed discussion of the use of Equation (4) in the revised version of our paper.**\n\n> **Q6) BraInCoRL conditioned on 9000 images**\n\nThis is a great suggestion. We have added an additional experiment to evaluate the performance of BraInCoRL with 9000 in-context samples using the CLIP backbone, for the four NSD subjects (S1, S2, S5, S7) in our main paper. Note that in this case, the model has not seen the subject we test on during training, and the model was trained on the other three subjects.\n\nAs shown in the table, the performance difference is less than $0.01$ explained variance difference across all subjects when compared to a fully trained model (which is fit on each subject's entire 9000-image training set). This means BraInCoRL achieves 94-99% of the fully trained model's performance. Crucially, BraInCoRL reaches this near-optimal performance while requiring no parameter updates or gradient-based optimization during the in-context inference. \n\n* Voxel-wise explained variance of BraInCoRL with the CLIP backbone compared with the fully trained reference model. The difference variance explained is less than 1% (higher is better).\n\n|Subject|S1|S2|S5|S7\n|-|-|-|-|-\n|Fully Trained|0.1765|0.1882|0.2310|0.1554\n|BraInCoRL|0.1667|0.1817|0.2225|0.1541\n\n> **Q7) Number of images clarification**\n\nIn the result on page 6, we intend to say that we can **utilize up to** each subject's 9000 unique images in the NSD dataset as the in-context support set. But during the evaluation experiment (such as shown in Table 1 caption), we just randomly sample a subset of e.g. 100 in-context images as the inference input. **We have clarified this in the revision.**\n\n> **Q8) Size of BOLD5000**\n\nThis discrepancy arises from the inherent nature of the BOLD5000 dataset itself. The NSD dataset involves up to $10,000$ images for each subject being viewed up to three times. In contrast, for BOLD5000, only approximately $100$ images were viewed more than three times.\n\nWe were advised by the BOLD5000 authors to only analyze the image subset with more than three repeats. Following our discussions with the BOLD5000 dataset authors, we follow the sampling strategy below: we randomly sample 20% of the BOLD5000 images as inference images, with the remaining 80% used as the in-context support set. \n> **Conclusion**\n\nWe are very grateful for the highly detailed and actionable suggestions. \nWe have included additional details for our method, and conducted additional experiments. We look forward to any additional questions.\n\n[1] Better models of human high-level visual cortex emerge from natural language supervision with a large and diverse dataset (2023).\n\n[2] Analyzing the scale operation of attention from the perspective of entropy invariance (2021). (see line 564 of main paper).\n\n[3] Overcoming a Theoretical Limitation of Self-Attention (2022).\n\n[4] Qwen technical report (2023).\n\n[5] The Llama 4 herd: The beginning of a new era of natively multimodal ai innovation (2025)."}, {"review_id": "1HohKJJ0nA", "reviewer": "Reviewer_CYvz", "summary": "The authors propose BraInCoRL as a model for predicting neuronal fMRI responses. The approach uses a transformer-based model with meta-learning across voxels and subjects, and few-shot in-context learning across stimuli without the need for finetuning. The authors show that the model generalizes across subjects, stimuli and recording acquisition parameters, outperforming anatomical and subject-wise baselines. They also explore attention patterns and carry out ablations as well as link visual cortex activity to natural language.", "questions": "1. The authors assume that the unobserved true response function is noiseless, i.e. all the noise is part of the measurement process. In reality there will be different noise components, some of which will be inherent to the brain response. Could it be useful to model these different sources in more detail?\n\n2. The visual response function is designed and required to be differentiable. How much of a limitation is this?\n\n3. Line 138: The authors used MSE optimization. Did they consider other error metrics?", "limitations": "Limitations of the study are discussed in a dedicated section and mention static vs. dynamic stimuli, as well as dataset diversity. Societal impacts are discussed in an appendix section. In my opinion, these sections address all reasonable points.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Overall, this study presents a solid modelling framework for fMRI responses. The methods are well evaluated and different applications are demonstrated. Minor weaknesses include lack of justifications of design decisions. My questions further elaborate on this.\n\nQuality:\nThe general problem is formalized cleanly. The authors perform a rigorous evaluation, making proper use of held-out data, and performing ablations. Baseline models are well selected and demonstrate the benefits of the proposed model and training, where no finetuning is required.\n\nClarity:\nThe paper is very well written. Figures and illustrations are clean and informative. The appendix also provides extensive methodological details on the proposed model and training. Minor points:\nEq. 4: redefines $c$ to represent context length. It might be better to use another symbol here to avoid confusion. Figure 7 should label the y-axis.\n\nSignificance:\nThe authors formulate the encoding problem as a meta-learning task, which addresses the problem that previous encoders were subject specific, requiring large individual datasets. The proposed model and training techniques demonstrate the generalization capabilities of modern models and training. Similar techniques might be applicable to other recording modalities or domains. The main immediate beneficiaries are practitioners using fMRI encoding models.\n\nOriginality:\nThere are previous studies on multi-subject approaches, as cited by the paper [30, 32], which do require finetuning. So the main advance of this work is the complete absence of needing finetuning. The authors adopt many techniques that were developed for LLMs, such as analysis-by-synthesis and making use of logit scaling for variable context length. These are adapted without proposing entirely new techniques but are novel in the context of fMRI encoders. Contrary to previous approaches, each voxel has a response function that is addressed in a meta-learning task. A transformer generates parameters of the visual response functions based on context tokens.", "was_revised": true, "final_justification": "I had minor concerns to begin with. The authors answered all of my questions and conducted additional analyses, confirming my initial assessment. For these reasons, I keep my score and recommend acceptance.", "rebuttal": "We appreciate your questions and concrete suggestions! We respond to specific questions below. We look forward to further discussion, and welcome additional comments.\n\n> **Q1) Stochastic visual response functions**\n\nYou are correct. Our current framework assumes a true response function that is noiseless, and is only affected by the current stimulus. Due to this assumption, our encoder outputs are also deterministic in nature. We adopted this setup because these conditions are commonly (sometimes implicitly) assumed in other fMRI encoder works.\n\nIf we want to construct an encoder that can capture the inherent distribution of a stochastic visual response function, we need to adopt a stochastic generative framework. There has been recent work on adopting diffusion models as fMRI encoders [1]; however their work currently does not capture the trial variability due to averaging across stimulus repeats. In such a framework, the variability across subjects, machines, stimuli, repeats, etc. can be modeled by introducing additional conditioning variables to the generative framework.\n\nWe agree this is an interesting future direction, and stochastic fMRI encoders could allow us to more accurately model the visual response.\n\n> **Q2) Differentiability of the visual response function**\n\nTo clarify, we **do not assume that the true biological visual response function is inherently differentiable**. Rather, we employ differentiable functions as computational approximations to model these unknown response patterns. We utilize differentiable functions because they can be optimized using modern gradient-based optimization techniques popularized by deep learning.\n\nModern neural encoding models in computational neuroscience commonly adopt this approach (such as [1,2]) because differentiable neural networks provide universal approximation capabilities, meaning they can approximate non-differentiable functions to high precision [3,4]. \n    \nAs shown in our **Experiments** section, BraInCoRL demonstrates that this differentiability constraint does not meaningfully limit our model's ability to capture biologically relevant response patterns. The successful identification of known category-selective regions, strong cross-dataset generalization, and attention patterns all indicate that differentiable approximations effectively capture the essential semantic selectivity of higher visual cortex. \n\n> **Q3) Choices of Losses**\n\nThis is a good question! Per your suggestion, we conduct an ablation study on the choice of different loss functions used to optimize the network on real neural data. \n\nOur experimental results show that MSE and L1 losses achieve similar performance across all context sizes, with minimal differences. This suggests that both metrics are equally effective for capturing voxelwise neural response patterns. We also explore the hybrid loss of 0.5 × MSE loss + 0.5 × (1 - cosine similarity), where the cosine loss is applied on ground truth voxel weights that are derived from ridge regression. We find this underperforms by 2-4% compared to MSE/L1. We believe this is due to the higher weighting on directional correctness of the weights, rather than optimizing for prediction error minimization. \n\nWe are motivated to adopt MSE since it corresponds to OLS regression commonly used for fMRI. MSE loss is also suitable because it provides optimal performance.\n\n* Voxel-wise explained variance of different training losses with the CLIP backbone for Subject 1 (higher is better).\n\n|  # In-Context Samples | 30| 70| 100| 300| 512| 700| 1000|\n|--------------------|---------------|---------------|---------------|---------------|---------------|---------------|---------------|\n| **MSE**| **0.0991** | **0.1283** | **0.1374** | **0.1571** | **0.1628** | **0.1650** | **0.1662** |\n| L1| 0.1005 | 0.1286 | 0.1375 | 0.1568 | 0.1626 | 0.1650 | 0.1650 |\n| MSE + CosSim| 0.0927 | 0.1213 | 0.1285 | 0.1498 | 0.1571 | 0.1593 | 0.1623 |\n\n> **Q4) Clarity of text**\n\nThank you for pointing out the typos and labels in our paper! We will update the paper according to your suggestions and those from other reviewers to improve the clarity of our upcoming revision.\n\n> **Conclusion**\n\nWe are grateful for your suggestions and detailed comments. We will incorporate your feedback and our corrections into the upcoming revision. We look forward to discussing with you further! \n\n\n[1] MindSimulator: Exploring Brain Concept Localization via Synthetic FMRI (2025).\n\n[2] A differentiable approach to multi-scale brain modeling (2024).\n\n[3] Multilayer feedforward networks are universal approximators (1989).\n\n[4] Approximation capabilities of multilayer feedforward networks (1991)."}]

Author rebuttal:

Meta-review: The paper presents a cross-subject fMRI encoding model based on in-context learning. The method, BraInCoRL, is valided against a simple Ridge regression baseline, but performance and trade-offs of the method are validated in quantitative and qualitative analysis throughout the paper. Post discussion, the reviewers remain unanimously positive about this work and recommend acceptance. I follow the recommendation, provided that the new results from the rebuttal phase are worked into the paper and the supplementary materials.
Decision: Accept (poster)