Title: Meta-Learning an In-Context Transformer Model of Human Higher Visual Cortex
Venue: NeurIPS 2025
Accepted: yes
Keywords: fMRI

brain

visual cortex

neuroscience

visual neuroscience

NeuroAI

visual selectivity

semantic selectivity

in-context learning
Abstract: Understanding functional representations within higher visual cortex is a fundamental question in computational neuroscience. While artificial neural networks pretrained on large-scale datasets exhibit striking representational alignment with human neural responses, learning image-computable models of visual cortex relies on individual-level, large-scale fMRI datasets. The necessity for expensive, time-intensive, and often impractical data acquisition limits the generalizability of encoders to new subjects and stimuli. **BraInCoRL** uses in-context learning to predict voxelwise neural responses from few-shot examples *without any additional finetuning* for novel subjects and stimuli. We leverage a transformer architecture that can flexibly condition on a variable number of in-context image stimuli, learning an inductive bias over multiple subjects. During training, we explicitly optimize the model for in-context learning. By jointly conditioning on image features and voxel activations, our model learns to directly generate better performing voxelwise models of higher visual cortex. We demonstrate that BraInCoRL consistently outperforms existing voxelwise encoder designs in a low-data regime when evaluated on entirely novel images, while also exhibiting strong test-time scaling behavior. The model also generalizes to an entirely new visual fMRI dataset, which uses different subjects and fMRI data acquisition parameters. Further, BraInCoRL facilitates better interpretability of neural signals in higher visual cortex by attending to semantically relevant stimuli. Finally, we show that our framework enables interpretable mappings from natural language queries to voxel selectivity.

Meta-Learning an In-Context Transformer Model of Human Higher Visual Cortex

Abstract

Understanding functional representations within higher visual cortex is a fundamental question in computational neuroscience. While artificial neural networks pretrained on large-scale datasets exhibit striking representational alignment with human neural responses, learning image-computable models of visual cortex relies on individual-level, large-scale fMRI datasets. The necessity for expensive, time-intensive, and often impractical data acquisition limits the generalizability of encoders to new subjects and stimuli. BraInCoRL uses incontext learning to predict voxelwise neural responses from few-shot examples without any additional finetuning for novel subjects and stimuli. We leverage a transformer architecture that can flexibly condition on a variable number of incontext image stimuli, learning an inductive bias over multiple subjects. During training, we explicitly optimize the model for in-context learning. By jointly conditioning on image features and voxel activations, our model learns to directly generate better performing voxelwise models of higher visual cortex. We demonstrate that BraInCoRL consistently outperforms existing voxelwise encoder designs in a low-data regime when evaluated on entirely novel images, while also exhibiting strong test-time scaling behavior. The model also generalizes to an entirely new visual fMRI dataset, which uses different subjects and fMRI data acquisition parameters. Further, BraInCoRL facilitates better interpretability of neural signals in higher visual cortex by attending to semantically relevant stimuli. Finally, we show that our framework enables interpretable mappings from natural language queries to voxel selectivity. Our code and model weights are publicly available at https://github.com/leomqyu/BraInCoRL .

1 Introduction

Human visual cortex transforms raw sensory input into behaviorally-relevant representations of the world. While early visual areas are characterized by retinotopic organization and selective tuning to simple features such as edges and orientation gradients [1, 2, 3, 4, 5], higher-order visual areas demonstrate selectivity to more abstract semantics and categories. While this functional organization is largely consistent across individuals at a coarse scale, the spatial distribution and finegrained semantic selectivity within visual cortex varies due to structural differences, developmental experience, and life-long learning [6, 7, 8, 9, 10, 11]. Such functional inter-subject differences pose a fundamental challenge in constructing generalizable models of higher visual cortex that can adapt to subject-specific neural organization without exhaustive data collection for every individual.

Recent advances in deep learning offer a promising avenue for addressing this challenge. Vision models pretrained on large-scale image datasets not only achieve strong object recognition performance, but also recapitulate hierarchical processing patterns observed in biological vision [12, 13, 14]. While these models may encapsulate some universal principles of visual processing [15], they do not inherently account for individual differences in cortical organization. To close the gap between artificial and biological systems, researchers have developed image-computable fMRI encoders models that predict brain activity from visual stimuli [16]. These encoders typically regress image features onto voxelwise brain responses using subject-specific data, acting as computational probes of visual processing. Unfortunately, current approaches require many hours of costly fMRI scans per subject to fit these mappings - a prohibitive bottleneck for scalability to new populations, stimuli, and tasks, especially in clinical settings where collecting large amounts of data is difficult.

We bridge this gap with BraInCoRL ( Bra in In -Co ntext R epresentation L earning), a transformerbased framework that meta-learns to predict subject-specific neural responses from provided examples. Inspired by language models that adapt to new tasks in-context, our approach treats voxel encoding as a function inference problem: given a handful of stimulus-response pairs from a new individual and novel stimuli, BraInCoRL constructs a voxelwise encoding model without any further training. By jointly optimizing for in-context learning across diverse subjects and stimuli, our model discovers shared functional principles of higher visual cortex that generalize to new subjects and stimuli represented by only a small amount of data. We illustrate the problem we are tackling in Figure 1.

We demonstrate that BraInCoRL: (1) Outperforms existing voxelwise encoder models in the low-data regime on novel visual stimuli while exhibiting strong generalization with increasing context. (2) Can generalize to new experiments with different scanning parameters. (3) Through analysis of attention values, learns to rely on images that are reflective of the category selected for in each region. (4) When paired with features from contrastive image-language models, facilitates zero-shot natural language-based characterization of cortical selectivity, enabling interpretable, finer-grained query-driven functional mapping.

2 Related work

Computational Encoding and Decoding Models for Visual Cortex. Computational modeling of neural data often involves two complementary approaches: encoding models that map from stimuli to neural activations, and decoding models that map from neural data to stimuli [16, 17, 18, 19, 20, 21, 22, 23, 24]. The development of both approaches has been facilitated by advances in machine learning models. For encoding models, the dominant approach is to combine pretrained deep feature extractors with linear voxelwise weights [25, 26, 27, 28, 29, 30]. More recent approaches have proposed to leverage transformers [31, 32] to learn the relationship between brain regions of a single subject. Most similar to our framework is the pioneering work by Adeli et al. [31] and Beliy & Wasserman et al. [33] which uses an auto-decoder based transformer network for multi-subject voxelwise encoding; However these approaches still require fine-tuning for novel subjects. More generally, encoders have been used to investigate the coding properties in higher-order visual areas [34, 35, 36, 37, 38, 39, 40, 41]. Encoders have been further combined with imagegeneration models [42, 43, 44, 45, 46, 47, 48, 49] or language-generation models [50, 51] to explore semantic selectivity. Recent progress on large generative models has enabled stimulus reconstruction from fMRI, EEG, and MEG signals for images [52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63], videos [64, 65, 66, 67, 68, 69, 70], and language/audio [71, 72, 73, 74, 75, 76, 77].

Representational Organization of Visual Cortex. Human visual cortex exhibits a hierarchical organization from primary visual to higher-order visual areas. The higher visual cortex is characterized by a tiling of semantically specialization. Approaches using functional localizers have identified category selective regions in higher visual that are responsive to faces [78, 79, 80, 81], places [82, 83], bodies [84], objects [85, 86], food [87, 88, 89], and words [90, 91]. While the spatial location of these broad category-selective regions are generally consistent across subjects [92], significant inter-individual variability exists in their anatomical location, spatial extent, and response profiles [6, 7, 8, 9, 93, 94, 95, 96, 97]. Accurately characterizing visual processing in higher-order visual areas necessitates subject-specific encoding models that capture individual diversity [98].

Meta-Learning and In-Context Learning. Our work builds upon meta-learning and in-context learning (ICL). Meta-learning trains models to "learn to learn" from a distribution of tasks, enabling

Space of response functions

(differs by voxel & subject)

(a)

Expl. Variance

# In-Context Samples

5.2x

1.2x

Fully Trained

BraInCoRL

Ridge Regression

0.0

0.7

Activations

Unobserved

True Response

Function

?

Other dims

Noisy

Measurement

Novel Stimuli

(c)

Measured

Visual stimuli

seen by subject

Input

Subj 1

Subj 2

Subj 3

Meta-training tasks

Training-free in-context adaptation to novel subject

New stimuli

New response

Model

Input

BraInCoRL

(b)

Voxel-wise function

estimation

Responses

(d)

。。。

BralnCoRL

0.02

■

Fully Trained

0

400

600

1000

BralnCoRL

RidgeBaseline

Fully Trained

400

600

800

1000

quick adaptation to new tasks with few examples, via methods like meta-optimization [99, 100, 101] or metric-based approaches [102]. More recently, ICL has emerged as a powerful capability in Large Language Models [103, 104], where models adapt to new tasks at inference time solely based on examples provided in their prompt without any parameter updates [105, 106]. This has led to hypotheses that ICL is an emergent form of implicit meta-learning, where transformers effectively learn underlying learning algorithms during pre-training [107, 108]. Our goal is to learn the structure of functions that map between visual stimuli and voxelwise brain response. Our framework combines meta-training (across voxels and subjects) and in-context learning (across stimuli) to enable training free adaptation to novel subjects.

3 Methods

Our proposed framework leverages meta-learning and uses few-shot, in-context examples for voxelwise prediction of unseen stimuli (Figure 2). Critically, for unseen subjects, this approach does not require any additional finetuning. We achieve this by treating the mapping function from visual stimuli and the response of individual voxels as a set of meta-training tasks. This voxelwise approach is in line with higher-order visual areas being described by a multitude of functionally diverse voxels, which we randomly sample during training.

3.1 Motivation and Problem Definition

There is substantial inter-subject anatomical and functional variability in the higher visual cortex among humans. Consequently, while one can learn per-subject image-computable encoders that map image features to brain responses with high predictive accuracy, these models require large amounts

In-Context Stimuli

Per-Image Non-Linear Feature Extractor

In-Context Conditioned

Higher Visual Cortex Transformer

Voxelwise

Encoder

Novel Stimuli

?

ω

Image

Embedding

Voxel

Activation

Generated

Encoder Weights

ω

of within-subject data and do not exploit information across subjects. To account for this variability across individuals, we design our framework to treat individual voxels as the fundamental unit of modeling. Importantly, our method does not assume any overlap in stimuli across subjects, yet still enables us to take advantage of multi-subject training data.

We formalize this problem by assuming an image j is represented as RGB tensor I j ∈ R H × W × 3 . Given an image j and a human subject k , there is a 1D array of voxel activations (beta values) from higher visual cortex: ( β 1 , β 2 , ..., β N k ) j = B j,k ∈ R 1 × N k , where the number of voxels N will differ between subjects.

Given a new subject not seen during training, we have a small set of n seen images ( I 1 , I 2 , ..., I n ) and measured brain responses ( B 1 , B 2 , ..., B n ) for this new subject. Our goal is to estimate the brain response to an arbitrary new image I novel .

3.2 Meta-Learning an In-Context Transformer

Image-computable encoders that map from images to brain responses for a single subject k are typically considered as a function f k ( I ) ⇒ B , and jointly model the entire visual cortex. While powerful, this approach cannot be easily extended to the multi-subject scenario, where test-time individuals may have functional and anatomical differences that are not known during training. In contrast, BraInCoRL considers each voxel v to have a unique and unobserved visual response function f k,v ( I ) ⇒ β v . Voxels can be from many different subjects. During training, we consider each voxel's response function to be a meta-training task, where each task is effectively specified by input images and voxel response pairs. In order to facilitate training-free adaptation on new subjects, we utilize in-context learning across stimuli enabled by a transformer backbone.

For a single voxel we define a support set of p images and neural responses { ( x 1 , β 1 ) , ( x 2 , β 2 ) , ..., ( x p , β p ) } , where x i ∈ R m is the image embedding vector extracted by a frozen image feature extractor ϕ ( · ) , i.e., x i = ϕ ( I i ) , and β i ∈ R is the voxel's response observed for image I i . Each pair is concatenated to form context tokens c i = [ x i ; β i ] , and the full context is defined as { c 1 , . . . , c p } . Unlike traditional in-context inference in large language models, where there is a query concatenated to the end of the context, we avoid expensive per-voxel inference by directly generating the parameters for the voxelwise visual response function. During training, we optimize the BraInCoRL transformer T with parameters θ such that it outputs voxel response function f with parameters ω :

Since T and f are differentiable, we optimize θ to maximize the likelihood of observing β given I :

In practice, we use mean-squared-error and gradient based mini-batch optimization.

3.3 Test-time Context Scaling

At test time, when we encounter a new subject, we assume we have access to a small set of novel images and the corresponding brain responses - we want to predict a voxelwise encoder. While our voxelwise parameterization successfully resolves the challenge of unknown test-time anatomy and geometry, the challenge of unknown test-time context size remains. Unlike transformers in language models, where the output is dependent on the order of the samples, we want our model to be explicitly invariant to input order. In order to facilitate variable length context, we utilize logit scaling [109, 110, 111]. Assuming a query/key ( q, k ) with d k features and a length l context:

We find this method effectively enables context scaling when trained with variable length context. While the hypernetwork could, in principle, parameterize any class of neural encoders (e.g., MLPs, convolution, attention layers), prior studies utilizing brain data have largely used linear parameterizations that map from deep network features to voxel responses [13, 14], and find that such a choice offers high performance and interpretability. Given features x ∈ R 1 × q from a pretrained neural network x = ϕ ( I ) , we adopt the same setup and predict the final voxel response:

A more detailed description on the test-time context scaling technique is provided in Appendix A.4.

4 Experiments

We utilize BraInCoRL to generate encoder weights in a low-data regime. We start by describing our experiment setup. First, we evaluate the effectiveness of BraInCoRL on novel subjects where there is zero overlap between the training dataset and the evaluated subject's in-context stimulus . We also evaluate our framework where data from novel subjects are collected from a completely different scanner and protocol . Second, we explore the attention pattern across stimuli for different ROIs, and perform ablations to explore the need for test-time in-context stimulus diversity. Third, we show that our method enables natural language characterizations of the visual cortex using very little data.

4.1 Setup

Dataset. We primarily perform experiments with the Natural Scenes Dataset (NSD) [112], but then validate with the BOLD5000 dataset [113]. Both are large-scale neural datasets: NSD is the largest 7T fMRI image viewing dataset available, where eight subjects each viewed ∼ 10 , 000 images; BOLD5000 is a 3T dataset, where four subjects each viewed ∼ 5000 images. In NSD each image was viewed up to three times, while in BOLD5000 only a subset of images were viewed up to four times. For NSD, of the eight subjects, four subjects (S1, S2, S5, S7) completed scanning and are the focus of our analysis in the main paper. The results of other subjects are presented in the supplemental

(b)

(a)

Subj1

Subj2

Subj5

Subj7

Expl. Variance

(c)

CLIP

DINO

SigLIP

Subj2

Subj7

# In-Context Samples

# In-Context Samples

7.1x

1.2x

10x

1.2x

Subj5

# In-Context Samples

10x

1.2x

Expl. Variance

Fully Trained

y=x

y=x

y=x

0.60

0.40

0.20

0.00

materials. For each subject, ∼ 9 , 000 images are unique to each other, while ∼ 1 , 000 are viewed by all eight subjects. The NSD stimuli were sourced from the MS COCO dataset (as were a subset of the BOLD5000 stimuli). Unless otherwise noted, we perform our analysis in subject native volume space ( func1pt8mm ) for NSD, where the voxelwise betas are z-scored within each session then averaged across repeats of the same stimulus. In order to rigorously evaluate BraInCoRL for a given subject, we use the 3 × 9,000 unique images viewed by the other three subjects as the meta-training data. During the ROI-wise evaluation for NSD, we follow prior work [48] and apply a t -statistic cutoff of t > 2 by leveraging independent functional localizer data provided with the dataset to threshold the originally broad definitions. During quantitative evaluation, we follow prior work [114] and apply a voxel quality cutoff of ncsnr > 0 . 2 . For BOLD5000, we use a model trained on four NSD subjects (S1, S2, S5, S7). Following the suggestion of BOLD5000 authors, we only model stimuli with 4 repeats and apply a cutoff of ncsnr > 0 . 3 . Voxel responses are averaged across repeats.

Training and evaluation. Our training process takes inspiration from LLM based training setups, and we adopt a three stage process - pretraining, context extension, and supervised fine tuning. In the pretraining stage, we use an analysis-by-synthesis approach without relying on any (real) subject data. We artificially construct a large number of voxels with synthesized weights. We derive synthetic voxel responses with normally distributed noise using these synthesized weights and train our model using a fixed context size of 500. In the second stage, we randomly sample the context size from Uniform (30 , 500) which allows the model to acquire length robustness. Finally, in the finetuning stage, the model is trained on real fMRI data using the subject-specific beta values,

(a)

(b)

UMAP1

UMAP2

UMAP3

Left

Hemisphere

Right Hemisphere

Anterior

Anterior

Superior

(c)

S2

S5

S7

EBA

EBA

PPA

PPA

RSC

RSC

OPA

OPA

FFA-2

FFA-1

FFA-1

aTL-

faces

aTL-

faces

Food

region

Food

region

enabling adaptation to biologically grounded neural responses. All our evaluation experiments are performed on novel subjects that are unseen by the model during training, with exception of (Figure 3b) no heldout ('no HO') ablation study.

4.2 Effectiveness of In-Context Higher Visual Cortex Encoders

On NSD. In this experiment, we evaluated BraInCoRL using each subject's 9,000 unique images as the in-context support set and the shared 1,000 images as the test set. For each evaluation, we randomly sampled training images from the subject-specific in-context support set and test images from the shared test set. Explained variance statistics averaged over category-selective ROIs are reported in Table 1. We compare BraInCoRL using just 100 test images against within-subject ridge regression baseline trained on 100 and 300 support-set images of the test-subject, with the regularization strength selected via 5-fold cross-validation over [10 -3 , 10 -2 , . . . , 10 8 ] . Remarkably, BraInCoRL with only 100 images nearly matches the performance of the fully supervised reference model that is trained by gradient descent on each subject's entire within-subject support-set of 9,000 images until convergence. We also evaluate an anatomical FsAverage baseline which aligns each training subject's anatomy to a common template and projects the average response onto novel subjects for prediction. While this baseline benefits from a strong anatomical prior, it is outperformed by BraInCoRL, which directly adapts to each subject's unique neural responses with higher efficiency.

To evaluate test-time behavior, we assess how performance scales with increasing in-context support size. BraInCoRL consistently outperforms within-subject ridge regression and more efficiently approaches the fully trained reference model (Figure 1c for subject 1 and Figure 3a for subject 2, 5, 7). Moreover, we conduct ablations by evaluating a BraInCoRL model trained without holding out the test subject's support images and a BraInCoRL model with only pretraining. Results confirm

CSI1

CSI3

# In-Context Samples

# In-Context Samples

2.7x

1.3x

3.9x

1.4x

# In-Context Samples

1.3x

Pearson

r

4.2x

10

05

1

BralnCoRL

RidgeBaseline

00

20

40

60

80

20

40

20

40

POCCW

Face Voxels

Food Voxels

STOP

ORAMA

POOPING

that finetuning with real neural data boosts performance and that BraInCoRL can generalize well to previously unseen images without overfitting (Figure 3b). Additionally, we observe high voxelwise explained variance correlation between BraInCoRL and the fully trained reference model across multiple backbones (Figure 3c). Finally, we apply UMAP to the BraInCoRL predicted responsefunction weights, revealing clear semantic clustering across higher visual areas (Figure 4) that correspond with known visual regions.

On BOLD5000. We validate generalization on the BOLD5000 dataset in Figure 5. BOLD5000 has many differences with NSD and represents the challenge of cross-site generalization that is the main objective of our method. BOLD5000 was collected on a 3T scanner with a different stimulus presentation time, a slow-event related trial structure (10s inter-trial interval), different images and image datasets, a different voxel size (2mm isotropic), and different subjects. BraInCoRL achieves higher voxelwise Pearson correlations than within-subject ridge regression. Moreover, results remain consistent across different subjects, demonstrating the robustness and reliability of our method.

4.3 Semantic Discovery through Text-Image Embedding Alignment

To better understand how BraInCoRL leverages in-context examples, we analyze its internal attention mechanisms to identify images that strongly influence voxel predictions in category-selective regions. In Figure 6, we examine attention weights from BraInCoRL's final attention layer to determine the top-contributing images for each cortical region. The visualized images with the highest attention scores closely align with known semantic preferences of the respective cortical regions.

However, Figure 7 reveals a counterintuitive finding regarding the semantic specificity of in-context support sets. We systematically vary the specificity of the 100-image sets provided to the model, ranging from highly relevant to random selections. Selections are determined via the first text-prompt in each category (see Appendix). We observe that randomly selected images lead to better predictive performance compared to sets composed solely of highly relevant images. This suggests that overly specific context sets may limit the generalization capabilities of the encoder system, and diverse, less semantically constrained images provide richer context for learning robust voxel representations.

0.20

0.15

0.10

0.05

0.00

0.30

0.20

0.10

0.00

Food

Words

Places

Faces

0

S1

S2

S5

S7

S1

S2

S5

S7

S1

S2

S5

S7

S1

S2

S5

S7

Random

Best 201-300

Best 301-400

Best 100

Best 101-200

Expl. Variance

Expl. Variance

vmin

vmax

'A photo of a bedroom'

'A photo of words'

'A photo of a torso'

'A photo of a person's face'

probing

True

t

-statistics

4.4 Characterizing higher visual cortex with text embeddings to images

In this experiment, we investigate the capability of BraInCoRL to enable interpretable, query-driven functional mapping using natural language prompts.

In Figure 8, we demonstrate that natural language prompts can be effectively mapped to predicted voxel activations. For each category selective region, we convert the corresponding natural language prompt into a CLIP embedding and project it into the image feature space to directly predict voxel activations. The resulting activation maps closely match expected t -statistics, reflecting BraInCoRL's ability to support intuitive, language-driven cortical queries.

In the second analysis (Table 2), we quantitatively assess the accuracy of prompt-driven activation predictions. We measure the fraction of voxels within each category-selective region whose peak predicted activation aligns with the category indicated by the natural language query. Results confirm that BraInCoRL paired with language embeddings achieves a high level of alignment between predicted voxel selectivity and query semantics across multiple subjects. The predictions for word-selective voxels were notably less accurate. We hypothesize this discrepancy arises from the developmental and experiential variability inherent to the formation of word-selective regions, as these areas form predominantly through individualized learning experiences, such as reading and linguistic exposure, leading to greater inter-subject variability.

5 Discussion

Limitations and Future Work. Here we have shown that meta-learning an in-context model of higher visual cortex can yield high performance and strong data efficiency gains, outperforming anatomical (FsAverage) and subject-wise baselines on novel subjects. Our work currently focuses on static natural images, and extensions to dynamic stimuli would likely require a rethinking of the encoder backbone and network structure. Further, while we show strong generalization across scanners and utilize the largest fMRI dataset that is NSD, there may still be limitations in dataset diversity [115]. Collection of larger and more diverse fMRI datasets will help mitigate this issue.

Conclusion. We introduce a foundation model that serves as an fMRI encoder, mapping from natural images to voxelwise activations. We demonstrate that our method can adapt without any finetuning to new stimuli, subjects, scanners, and scanning protocols. Our model achieves this by meta-learning across voxels from different subjects, and performing in-context learning across stimuli. Our approach has significant data-efficiency and performance gains over baseline methods, and has the potential to help understand cortical structure in data-constrained environments.

References

Stephen W Kuffler. Discharge patterns and functional organization of mammalian retina. Journal of neurophysiology , 16(1):37-68, 1953.

David H Hubel and Torsten N Wiesel. Receptive fields, binocular interaction and functional architecture in the cat's visual cortex. The Journal of physiology , 160(1):106, 1962.

Joseph J Atick and A Norman Redlich. What does the retina know about natural scenes? Neural computation , 4(2):196-210, 1992.

Jan Verweij, Eric P Hornstein, and Julie L Schnapf. Surround antagonism in macaque cone photoreceptors. Journal of Neuroscience , 23(32):10249-10257, 2003.

Kalanit Grill-Spector and Rafael Malach. The human visual cortex. Annu. Rev. Neurosci. , 27(1): 649-677, 2004.

Michael J Tarr and Isabel Gauthier. FFA: a flexible fusiform area for subordinate-level visual processing automatized by expertise. Nature neuroscience , 3(8):764-769, 2000.

Isabel Gauthier, Pawel Skudlarski, John C Gore, and Adam W Anderson. Expertise for cars and birds recruits brain areas involved in face recognition. Nature neuroscience , 3(2):191-197, 2000.

Roel M Willems, Marius V Peelen, and Peter Hagoort. Cerebral lateralization of face-selective and body-selective visual areas depends on handedness. Cerebral cortex , 20(7):1719-1725, 2010.

Qing Cai, Lise Van der Haegen, and Marc Brysbaert. Complementary hemispheric specialization for language production and visuospatial attention. Proceedings of the National Academy of Sciences , 110(4):E322-E330, 2013.

Philippe Pinel, Christophe Lalanne, Thomas Bourgeron, Fabien Fauchereau, Cyril Poupon, Eric Artiges, Denis Le Bihan, Ghislaine Dehaene-Lambertz, and Stanislas Dehaene. Genetic and environmental influences on the visual word form and fusiform face areas. Cerebral Cortex , 25 (9):2478-2493, 2015.

Zeynep M Saygin, David E Osher, Elizabeth S Norton, Deanna A Youssoufian, Sara D Beach, Jenelle Feather, Nadine Gaab, John DE Gabrieli, and Nancy Kanwisher. Connectivity precedes function in the development of the visual word form area. Nature neuroscience , 19(9):1250-1255, 2016.

Daniel LK Yamins, Ha Hong, Charles F Cadieu, Ethan A Solomon, Darren Seibert, and James J DiCarlo. Performance-optimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the national academy of sciences , 111(23):8619-8624, 2014.

Aria Yuan Wang, Kendrick Kay, Thomas Naselaris, Michael J Tarr, and Leila Wehbe. Better models of human high-level visual cortex emerge from natural language supervision with a large and diverse dataset. Nat Mach Intell , 5:1415-1426, 2023.

Colin Conwell, Jacob S Prince, Kendrick N Kay, George A Alvarez, and Talia Konkle. What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines? BioRxiv , pages 2022-03, 2022.

Zirui Chen and Michael F. Bonner. Universal dimensions of visual representation, 2024.

Thomas Naselaris, Kendrick N Kay, Shinji Nishimoto, and Jack L Gallant. Encoding and decoding in fmri. Neuroimage , 56(2):400-410, 2011.

Yukiyasu Kamitani and Frank Tong. Decoding the visual and subjective contents of the human brain. Nature neuroscience , 8(5):679-685, 2005.

Kenneth A Norman, Sean M Polyn, Greg J Detre, and James V Haxby. Beyond mind-reading: multi-voxel pattern analysis of fmri data. Trends in cognitive sciences , 10(9):424-430, 2006.

Kuan Han, Haiguang Wen, Junxing Shi, Kun-Han Lu, Yizhen Zhang, Di Fu, and Zhongming Liu. Variational autoencoder: An unsupervised model for encoding and decoding fmri activity in visual cortex. NeuroImage , 198:125-136, 2019.

Katja Seeliger, Umut Güçlü, Luca Ambrogioni, Yagmur Güçlütürk, and Marcel AJ van Gerven. Generative adversarial networks for reconstructing natural images from brain activity. NeuroImage , 181:775-785, 2018.

Guohua Shen, Tomoyasu Horikawa, Kei Majima, and Yukiyasu Kamitani. Deep image reconstruction from human brain activity. PLoS computational biology , 15(1):e1006633, 2019.

Ziqi Ren, Jie Li, Xuetong Xue, Xin Li, Fan Yang, Zhicheng Jiao, and Xinbo Gao. Reconstructing seen image from brain activity by visually-guided cognitive representation and adversarial learning. NeuroImage , 228:117602, 2021.

Yuqin Dai, Zhouheng Yao, Chunfeng Song, Qihao Zheng, Weijian Mai, Kunyu Peng, Shuai Lu, Wanli Ouyang, Jian Yang, and Jiamin Wu. Mindaligner: Explicit brain functional alignment for cross-subject visual decoding from limited fmri data, 2025. URL https://arxiv.org/abs/ 2502.05034 .

Alessandro T Gifford, Benjamin Lahner, Pablo Oyarzo, Aude Oliva, Gemma Roig, and Radoslaw M Cichy. What opportunities do large-scale visual neural datasets offer to the vision sciences community? Journal of Vision , 24(10):152-152, 2024.

Serge O Dumoulin and Brian A Wandell. Population receptive field estimates in human visual cortex. Neuroimage , 39(2):647-660, 2008.

Umut Güçlü and Marcel AJ Van Gerven. Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. Journal of Neuroscience , 35(27): 10005-10014, 2015.

David Klindt, Alexander S Ecker, Thomas Euler, and Matthias Bethge. Neural system identification for large populations separating 'what' and 'where'. Advances in neural information processing systems , 30, 2017.

Michael Eickenberg, Alexandre Gramfort, Gaël Varoquaux, and Bertrand Thirion. Seeing it all: Convolutional network layers map the function of the human visual system. NeuroImage , 152: 184-194, 2017.

Haiguang Wen, Junxing Shi, Yizhen Zhang, Kun-Han Lu, Jiayue Cao, and Zhongming Liu. Neural encoding and decoding with deep learning for dynamic natural vision. Cerebral cortex , 28(12):4136-4160, 2018.

Guy Gaziv, Roman Beliy, Niv Granot, Assaf Hoogi, Francesca Strappini, Tal Golan, and Michal Irani. Self-supervised natural image reconstruction and large-scale semantic classification from brain activity. NeuroImage , 254:119121, 2022.

Hossein Adeli, Sun Minni, and Nikolaus Kriegeskorte. Predicting brain activity using transformers. bioRxiv , pages 2023-08, 2023.

Guangyin Bao, Qi Zhang, Zixuan Gong, Zhuojia Wu, and Duoqian Miao. Mindsimulator: Exploring brain concept localization via synthetic fmri. arXiv preprint arXiv:2503.02351 , 2025.

Roman Beliy, Navve Wasserman, Amit Zalcher, and Michal Irani. The wisdom of a crowd of brains: A universal brain encoder. arXiv preprint arXiv:2406.12179 , 2024.

Meenakshi Khosla and Leila Wehbe. High-level visual areas act like domain-general filters with strong selectivity and functional specialization. bioRxiv , pages 2022-03, 2022.

Meenakshi Khosla, Keith Jamison, Amy Kuceyeski, and Mert Sabuncu. Characterizing the ventral visual stream with response-optimized neural encoding models. Advances in Neural Information Processing Systems , 35:9389-9402, 2022.

Cory Efird, Alex Murphy, Joel Zylberberg, and Alona Fyshe. What's the opposite of a face? finding shared decodable concepts and their negations in the brain. arXiv e-prints , pages arXiv2405, 2024.

Huzheng Yang, James Gee, and Jianbo Shi. Brain decodes deep nets. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 23030-23040, 2024.

Huzheng Yang, James Gee, and Jianbo Shi. Alignedcut: Visual concepts discovery on brainguided universal feature space. arXiv preprint arXiv:2406.18344 , 2024.

Andrew F Luo, Jacob Yeung, Rushikesh Zawar, Shaurya Dewan, Margaret M Henderson, Leila Wehbe, and Michael J Tarr. Brain mapping with dense features: Grounding cortical semantic selectivity in natural images with vision transformers. arXiv preprint arXiv:2410.05266 , 2024.

Gabriel H. Sarch, Michael J. Tarr, Katerina Fragkiadaki, and Leila Wehbe. Brain dissection: fmri-trained networks reveal spatial selectivity in the processing of natural images. bioRxiv , 2023. doi: 10.1101/2023.05.29.542635. URL https://www.biorxiv.org/content/early/ 2023/11/20/2023.05.29.542635 .

Alexander Lappe, Anna Bognár, Ghazaleh Ghamkahri Nejad, Albert Mukovskiy, Lucas Martini, Martin Giese, and Rufin Vogels. Parallel backpropagation for shared-feature visualization. Advances in Neural Information Processing Systems , 37:22993-23012, 2024.

Edgar Y Walker, Fabian H Sinz, Erick Cobos, Taliah Muhammad, Emmanouil Froudarakis, Paul G Fahey, Alexander S Ecker, Jacob Reimer, Xaq Pitkow, and Andreas S Tolias. Inception loops discover what excites neurons most using deep predictive models. Nature neuroscience , 22 (12):2060-2065, 2019.

Pouya Bashivan, Kohitij Kar, and James J DiCarlo. Neural population control via deep image synthesis. Science , 364(6439):eaav9436, 2019.

Carlos R Ponce, Will Xiao, Peter F Schade, Till S Hartmann, Gabriel Kreiman, and Margaret S Livingstone. Evolving images for visual neurons using a deep generative network reveals coding principles and neuronal preferences. Cell , 177(4):999-1009, 2019.

N Apurva Ratan Murty, Pouya Bashivan, Alex Abate, James J DiCarlo, and Nancy Kanwisher. Computational models of category-selective brain regions enable high-throughput tests of selectivity. Nature communications , 12(1):5540, 2021.

Zijin Gu, Keith Wakefield Jamison, Meenakshi Khosla, Emily J Allen, Yihan Wu, Ghislain St-Yves, Thomas Naselaris, Kendrick Kay, Mert R Sabuncu, and Amy Kuceyeski. NeuroGen: activation optimized image synthesis for discovery neuroscience. NeuroImage , 247:118812, 2022.

Paweł A Pierzchlewicz, Konstantin F Willeke, Arne F Nix, Pavithra Elumalai, Kelli Restivo, Tori Shinn, Cate Nealley, Gabrielle Rodriguez, Saumil Patel, Katrin Franke, et al. Energy guided diffusion for generating neurally exciting images, 2023.

Andrew F Luo, Margaret M Henderson, Leila Wehbe, and Michael J Tarr. Brain diffusion for visual exploration: Cortical discovery using large scale generative models. arXiv preprint arXiv:2306.03089 , 2023.

Diego García Cerdas, Christina Sartzetaki, Magnus Petersen, Gemma Roig, Pascal Mettes, and Iris Groen. Brainactiv: Identifying visuo-semantic properties driving cortical selectivity using diffusion-based image manipulation. bioRxiv , pages 2024-10, 2024.

Andrew Luo, Margaret Marie Henderson, Michael J. Tarr, and Leila Wehbe. Brainscuba: Fine-grained natural language captions of visual cortex selectivity. In The Twelfth International Conference on Learning Representations , 2024. URL https://openreview.net/forum?id= mQYHXUUTkU .

Takuya Matsuyama, Shinji Nishimoto, and Yu Takagi. Lavca: Llm-assisted visual cortex captioning. arXiv preprint arXiv:2502.13606 , 2025.

Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. bioRxiv , pages 2022-11, 2022.

Zijiao Chen, Jiaxin Qing, Tiange Xiang, Wan Lin Yue, and Juan Helen Zhou. Seeing beyond the brain: Conditional diffusion model with sparse masked modeling for vision decoding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 22710-22720, 2023.

Yizhuo Lu, Changde Du, Dianpeng Wang, and Huiguang He. Minddiffuser: Controlled image reconstruction from human brain activity with semantic and structural diffusion. arXiv preprint arXiv:2303.14139 , 2023.

Furkan Ozcelik and Rufin VanRullen. Brain-diffuser: Natural scene reconstruction from fmri signals using generative latent diffusion. arXiv preprint arXiv:2303.05334 , 2023.

Adrien Doerig, Tim C Kietzmann, Emily Allen, Yihan Wu, Thomas Naselaris, Kendrick Kay, and Ian Charest. Semantic scene descriptions as an objective of human vision. arXiv preprint arXiv:2209.11737 , 2022.

Matteo Ferrante, Furkan Ozcelik, Tommaso Boccato, Rufin VanRullen, and Nicola Toschi. Brain captioning: Decoding human brain activity into images and text. arXiv preprint arXiv:2305.11560 , 2023.

Yulong Liu, Yongqiang Ma, Wei Zhou, Guibo Zhu, and Nanning Zheng. Brainclip: Bridging brain and visual-linguistic representation via clip for generic natural visual stimulus decoding from fmri. arXiv preprint arXiv:2302.12971 , 2023.

Weijian Mai and Zhijun Zhang. Unibrain: Unify image reconstruction and captioning all in one diffusion model from human brain activity. arXiv preprint arXiv:2308.07428 , 2023.

Paul S Scotti, Mihir Tripathy, Cesar Kadir Torrico Villanueva, Reese Kneeland, Tong Chen, Ashutosh Narang, Charan Santhirasegaran, Jonathan Xu, Thomas Naselaris, Kenneth A Norman, et al. Mindeye2: Shared-subject models enable fmri-to-image with 1 hour of data. arXiv preprint arXiv:2403.11207 , 2024.

Yohann Benchetrit, Hubert Banville, and Jean-Rémi King. Brain decoding: toward real-time reconstruction of visual perception. arXiv preprint arXiv:2310.19812 , 2023.

Dongyang Li, Chen Wei, Shiying Li, Jiachen Zou, Haoyang Qin, and Quanying Liu. Visual decoding and reconstruction via eeg embeddings with guided diffusion. arXiv preprint arXiv:2403.07721 , 2024.

Zhanqiang Guo, Jiamin Wu, Yonghao Song, Jiahui Bu, Weijian Mai, Qihao Zheng, Wanli Ouyang, and Chunfeng Song. Neuro-3d: Towards 3d visual decoding from eeg signals, 2024. URL https://arxiv.org/abs/2411.12248 .

Yu Zhu, Bo Lei, Chunfeng Song, Wanli Ouyang, Shan Yu, and Tiejun Huang. Multi-modal latent variables for cross-individual primary visual cortex modeling and analysis, 2024. URL https://arxiv.org/abs/2412.14536 .

Steffen Schneider, Jin Hwa Lee, and Mackenzie Weygandt Mathis. Learnable latent embeddings for joint behavioural and neural analysis. Nature , 617(7960):360-368, 2023.

Zijiao Chen, Jiaxin Qing, and Juan Helen Zhou. Cinematic mindscapes: High-quality video reconstruction from brain activity. Advances in Neural Information Processing Systems , 36: 24841-24858, 2023.

Zixuan Gong, Guangyin Bao, Qi Zhang, Zhongwei Wan, Duoqian Miao, Shoujin Wang, Lei Zhu, Changwei Wang, Rongtao Xu, Liang Hu, Ke Liu, and Yu Zhang. Neuroclips: Towards high-fidelity and smooth fMRI-to-video reconstruction. In The Thirty-eighth Annual Conference on Neural Information Processing Systems , 2024. URL https://openreview.net/forum? id=8qu52Fl1Dt .

Jacob Yeung, Andrew F Luo, Gabriel Sarch, Margaret M Henderson, Deva Ramanan, and Michael J Tarr. Neural representations of dynamic visual stimuli. arXiv preprint arXiv:2406.02659 , 2024.

Xuan-Hao Liu, Yan-Kai Liu, Yansen Wang, Kan Ren, Hanwen Shi, Zilong Wang, Dongsheng Li, Bao-Liang Lu, and Wei-Long Zheng. Eeg2video: Towards decoding dynamic visual perception from eeg signals. Advances in Neural Information Processing Systems , 37:72245-72273, 2024.

Camilo Fosco, Benjamin Lahner, Bowen Pan, Alex Andonian, Emilie Josephs, Alex Lascelles, and Aude Oliva. Brain netflix: Scaling data to reconstruct videos from brain signals. In European Conference on Computer Vision , pages 457-474. Springer, 2024.

Brian N Pasley, Stephen V David, Nima Mesgarani, Adeen Flinker, Shihab A Shamma, Nathan E Crone, Robert T Knight, and Edward F Chang. Reconstructing speech from human auditory cortex. PLoS biology , 10(1):e1001251, 2012.

Gaël Varoquaux, Pradeep Reddy Raamana, Denis A Engemann, Andrés Hoyos-Idrobo, Yannick Schwartz, and Bertrand Thirion. Assessing and tuning brain decoders: cross-validation, caveats, and guidelines. NeuroImage , 145:166-179, 2017.

Ludovic Bellier, Anaïs Llorens, Déborah Marciano, Aysegul Gunduz, Gerwin Schalk, Peter Brunner, and Robert T Knight. Music can be reconstructed from human auditory cortex activity using nonlinear decoding models. PLoS biology , 21(8):e3002176, 2023.

Subba Reddy Oota, Emin Çelik, Fatma Deniz, and Mariya Toneva. Speech language models lack important brain-relevant semantics. arXiv preprint arXiv:2311.04664 , 2023.

Hyejeong Jo, Yiqian Yang, Juhyeok Han, Yiqun Duan, Hui Xiong, and Won Hee Lee. Are eeg-to-text models working? arXiv preprint arXiv:2405.06459 , 2024.

Francis R Willett, Erin M Kunz, Chaofei Fan, Donald T Avansino, Guy H Wilson, Eun Young Choi, Foram Kamdar, Matthew F Glasser, Leigh R Hochberg, Shaul Druckmann, et al. A high-performance speech neuroprosthesis. Nature , 620(7976):1031-1036, 2023.

Sean L Metzger, Kaylo T Littlejohn, Alexander B Silva, David A Moses, Margaret P Seaton, Ran Wang, Maximilian E Dougherty, Jessie R Liu, Peter Wu, Michael A Berger, et al. A high-performance neuroprosthesis for speech decoding and avatar control. Nature , 620(7976): 1037-1046, 2023.

Justine Sergent, Shinsuke Ohta, and Brennan Macdonald. Functional neuroanatomy of face and object processing: a positron emission tomography study. Brain , 115(1):15-36, 1992.

Truett Allison, Gregory McCarthy, Anna Nobre, Aina Puce, and Aysenil Belger. Human extrastriate visual cortex and the perception of faces, words, numbers, and colors. Cerebral cortex , 4(5):544-554, 1994.

Gregory McCarthy, Aina Puce, John C Gore, and Truett Allison. Face-specific processing in the human fusiform gyrus. Journal of cognitive neuroscience , 9(5):605-610, 1997.

Nancy Kanwisher, Josh McDermott, and Marvin M Chun. The fusiform face area: a module in human extrastriate cortex specialized for face perception. Journal of neuroscience , 17(11): 4302-4311, 1997.

Geoffrey K Aguirre, John A Detre, David C Alsop, and Mark D'Esposito. The parahippocampus subserves topographical learning in man. Cerebral cortex , 6(6):823-829, 1996.

Russell Epstein and Nancy Kanwisher. A cortical representation of the local visual environment. Nature , 392(6676):598-601, 1998.

Paul E Downing, Yuhong Jiang, Miles Shuman, and Nancy Kanwisher. A cortical area selective for visual processing of the human body. Science , 293(5539):2470-2473, 2001.

Kalanit Grill-Spector. The neural basis of object perception. Current opinion in neurobiology , 13(2):159-166, 2003.

Rafael Malach, JB Reppas, RR Benson, KK Kwong, H Jiang, WA Kennedy, PJ Ledden, TJ Brady, BR Rosen, and RB Tootell. Object-related activity revealed by functional magnetic resonance imaging in human occipital cortex. Proceedings of the National Academy of Sciences , 92(18):8135-8139, 1995.

Meenakshi Khosla, N. Apurva Ratan Murty, and Nancy Kanwisher. A highly selective response to food in human visual cortex revealed by hypothesis-free voxel decomposition. Current Biology , 32:1-13, 2022.

Ian ML Pennock, Chris Racey, Emily J Allen, Yihan Wu, Thomas Naselaris, Kendrick N Kay, Anna Franklin, and Jenny M Bosten. Color-biased regions in the ventral visual pathway are food selective. Current Biology , 33(1):134-146, 2023.

Nidhi Jain, Aria Wang, Margaret M. Henderson, Ruogu Lin, Jacob S. Prince, Michael J. Tarr, and Leila Wehbe. Selectivity for food in human ventral visual cortex. Communications Biology 2023 6:1 , 6:1-14, 2 2023. ISSN 2399-3642. doi: 10.1038/s42003-023-04546-2.

Laurent Cohen, Stanislas Dehaene, Lionel Naccache, Stéphane Lehéricy, Ghislaine DehaeneLambertz, Marie-Anne Hénaff, and François Michel. The visual word form area: spatial and temporal characterization of an initial stage of reading in normal subjects and posterior split-brain patients. Brain , 123(2):291-307, 2000.

Stanislas Dehaene, Lionel Naccache, Laurent Cohen, Denis Le Bihan, Jean-François Mangin, Jean-Baptiste Poline, and Denis Rivière. Cerebral mechanisms of word masking and unconscious repetition priming. Nature neuroscience , 4(7):752-758, 2001.

Hans P Op de Beeck, Jennifer A Deutsch, Wim Vanduffel, Nancy G Kanwisher, and James J DiCarlo. A stable topography of selectivity for unfamiliar shape classes in monkey inferior temporal cortex. Cerebral Cortex , 18(7):1676-1694, 2008.

Job van den Hurk, Marc Van Baelen, and Hans P Op de Beeck. Development of visual category selectivity in ventral visual cortex does not require visual experience. Proceedings of the National Academy of Sciences , 114(22):E4501-E4510, 2017.

Golijeh Golarai, Alina Liberman, and Kalanit Grill-Spector. Experience shapes the development of neural substrates of face processing in human ventral temporal cortex. Cerebral Cortex , 27(2): bhv314, 2015.

Tina T Liu, Adrian Nestor, Mark D Vida, John A Pyles, Christina Patterson, Ying Yang, Fan Nils Yang, Erez Freud, and Marlene Behrmann. Successful reorganization of category-selective visual cortex following occipito-temporal lobectomy in childhood. Cell reports , 24(5):1113-1122, 2018.

Hans P Op de Beeck, Ineke Pillet, and J Brendan Ritchie. Factors determining where categoryselective areas emerge in visual cortex. Trends in cognitive sciences , 23(9):784-797, 2019.

Nooshin Abbasi, John Duncan, and Reza Rajimehr. Genetic influence is linked to cortical morphology in category-selective areas of visual cortex. Nature Communications , 11(1):709, 2020.

Alfonso Nieto-Castañón and Evelina Fedorenko. Subject-specific functional localizers increase sensitivity and functional resolution of multi-subject analyses. Neuroimage , 63(3):1646-1669, 2012.

Chelsea Finn, Pieter Abbeel, and Sergey Levine. Model-agnostic meta-learning for fast adaptation of deep networks. In International conference on machine learning , pages 1126-1135. PMLR, 2017.

Alex Nichol, Joshua Achiam, and John Schulman. On first-order meta-learning algorithms. arXiv preprint arXiv:1803.02999 , 2018.

Aravind Rajeswaran, Chelsea Finn, Sham M Kakade, and Sergey Levine. Meta-learning with implicit gradients. Advances in neural information processing systems , 32, 2019.

Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems , 30, 2017.

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems , 33:1877-1901, 2020.

Johannes Von Oswald, Eyvind Niklasson, Ettore Randazzo, João Sacramento, Alexander Mordvintsev, Andrey Zhmoginov, and Max Vladymyrov. Transformers learn in-context by gradient descent. In International Conference on Machine Learning , pages 35151-35174. PMLR, 2023.

Sewon Min, Mike Lewis, Luke Zettlemoyer, and Hannaneh Hajishirzi. Metaicl: Learning to learn in context. arXiv preprint arXiv:2110.15943 , 2021.

Julian Coda-Forno, Marcel Binz, Zeynep Akata, Matt Botvinick, Jane Wang, and Eric Schulz. Meta-in-context learning in large language models. Advances in Neural Information Processing Systems , 36:65189-65201, 2023.

Shivam Garg, Dimitris Tsipras, Percy S Liang, and Gregory Valiant. What can transformers learn in-context? a case study of simple function classes. Advances in Neural Information Processing Systems , 35:30583-30598, 2022.

Damai Dai, Yutao Sun, Li Dong, Yaru Hao, Shuming Ma, Zhifang Sui, and Furu Wei. Why can gpt learn in-context? language models implicitly perform gradient descent as meta-optimizers. arXiv preprint arXiv:2212.10559 , 2022.

Jianlin Su. Analyzing the scale operation of attention from the perspective of entropy invariance, Dec 2021. URL https://kexue.fm/archives/8823 .

David Chiang and Peter Cholak. Overcoming a theoretical limitation of self-attention. arXiv preprint arXiv:2202.12172 , 2022.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609 , 2023.

Emily J Allen, Ghislain St-Yves, Yihan Wu, Jesse L Breedlove, Jacob S Prince, Logan T Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, et al. A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence. Nature neuroscience , 25(1):116-126, 2022.

Nadine Chang, John A Pyles, Austin Marcus, Abhinav Gupta, Michael J Tarr, and Elissa M Aminoff. Bold5000, a public fMRI dataset while viewing 5000 visual images. Scientific Data , 6 (1):1-18, 2019.

Colin Conwell, Jacob S Prince, Kendrick N Kay, George A Alvarez, and Talia Konkle. A large-scale examination of inductive biases shaping high-level visual representation in brains and machines. Nature communications , 15(1):9383, 2024.

Ken Shirakawa, Yoshihiro Nagano, Misato Tanaka, Shuntaro C Aoki, Kei Majima, Yusuke Muraki, and Yukiyasu Kamitani. Spurious reconstruction from brain activity. arXiv preprint arXiv:2405.10078 , 2024.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The claims made in the abstract and introduction are reflected and addressed in the experiments section.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitation of the work is discussed in the discussion section.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [NA]

Justification: The paper does not include theoretical results.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The learning theory is described in the methods section. The model training details are discussed in the experiment setup section and in the supplemental material section. We will release the code and checkpoints before the meeting.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: All the data, code and checkpoints will be published before the meeting (with GitHub link included). Also, the instructions to reproduce experimental results are provided in the experimental results and supplemental material sections.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The major experimental settings including how to train and test the model is described in the experiments section. More details are provided in the supplemental material section.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The experiment results are accompanied by error bars in the experiments section and supplemental material section when available.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The details on computational resources are specified in the supplemental material section.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The points are discussed in the discussion section and supplemental material section.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: To the best of our knowledge, the paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: The creators or original owners of assets are properly credited and the license and terms of use are explicitly mentioned and properly respected in the main text and in the supplemental material.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: We release our model's code via our anonymous GitHub URL.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: No new data collection or crowdsourcing was performed. Our analyses use only the public BOLD5000 and NSD dataset.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: We did not collect any new human-subject data. All analyses are performed on the publicly available BOLD5000 and NSD fMRI datasets, which require no additional IRB review.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification: The core method development in this research does not involve LLMs as any important, original, or non-standard components.

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/LLM ) for what should or should not be described.

A Technical Appendices and Supplementary Material

Sections

Social impacts (Section A.1)

Implementation details (Section A.2)

Text prompts for category-selective regions (Section A.3)

A more detailed description of logit scaling (Section A.4)

Cortex prediction explained variance for different image encoding backbones and subjects (Section A.5)

Voxelwise performance across five category-selective regions for different image encoding backbones and subjects (Section A.6)

Voxelwise explained variance across varying support set sizes for more subjects, backbones and for pretrain-only models (Section A.7)

Impact of holding out the test subject's unique images during meta-training evaluated on more backbones (Section A.8)

Correlation of each backbone's predictions with fully trained activation predictions (Section A.9)

Voxelwise explained variance evaluation in BOLD5000 for more subjects and different backbones (Section A.10)

Dimensional reduction of predicted response function weights on more subjects (Section A.11)

Predicting cortical responses from natural language prompts on more subjects (Section A.12)

Voxelwise prompt classification accuracy for more subjects (Section A.13)

Additional evaluation of BraInCoRL on NSD dataset (Section A.14)

Additional evaluation of BraInCoRL on BOLD5000 dataset (Section A.15)

Evaluation of each training stage's contribution (Section A.16)

Performance of BraInCoRL conditioned on the full 9000-image set (Section A.17)

Evaluation on the choice of loss function during training (Section A.18)

Ablation on logit scaling (Section A.19)

A.1 Social impacts

Our work introduces BraInCoRL, a meta-learning framework that uses fMRI-measured voxel responses and trained visual encoders to perform in-context adaptation: given a small support set of image-response pairs, the model directly estimates each voxel's response-function parameters for novel stimuli. BraInCoRL's fusion of diverse image data and neural measurements uncovers data-driven principles of cortical organization beyond both traditional hypothesis-driven experiments and computational encoding models that require thousands of samples. Moreover, its alignment of neural responses with natural language prompts enables the generation of new hypotheses about semantic representation in visual cortex. As such, BraInCoRL may accelerate early diagnosis and monitoring of visual or neurological disorders via rapid, subject-specific cortical mapping; guide more efficient experimental design through optimized stimulus selection; deepen our understanding of semantic coding and inter-subject variability; and, when integrated with generative image models, open avenues for brain-guided stimulus synthesis and personalized neuroprosthetic and brain-computer interface development. While BraInCoRL offers significant potential for neuroscience and clinical applications, its reliance on fMRI datasets and computational infrastructure may limit accessibility for under-resourced research groups and raise privacy concerns if applied to sensitive neural data.

A.2 Implementation details

Network architecture. Our BraInCoRL model architecture comprises three main components:

Input projection. An input projection MLP layer is applied to each token individually, which maps the stimulus semantics and voxel activation into an embedding. In detail, we concatenate each image embedding with its scalar neural response and pass it through a single-layer MLP to align the two modalities into a unified internal feature space.

Transformer encoder. A stack of 20 self-attention layers integrates information across all support examples (plus learnable tokens), capturing contextual relationships and the relative importance of each in-context sample. We adopt best practices and utilize SwiGLU activation paired with pre-normalization in the attention block. We utilize 10 heads in each multi-head self attention.

Weight prediction. The [CLS] token from the final layer goes through an MLP to output a hyperweight which is used to parameterize the final encoder. In detail, the aggregated representation is fed through another single-layer MLP that outputs a weight vector, which is then used to linearly combine unknown-image embeddings to produce the final neural response predictions.

The CLIP-based variant (encoding dimension E = 512 ) contains approximately 97.2 M parameters; DINO ( E = 768 ) and SIGLIP ( E = 1152 ) variants comprise roughly 112 M and 130 M parameters, respectively.

Model training. Training is implemented in PyTorch on eight NVIDIA RTX 6000 Ada GPUs (48 GB each). We optimize using AdamW (decoupled weight decay 1 × 10 -4 ) with an initial learning rate of 1 × 10 -3 , which decays to 1 × 10 -5 via a ReduceLROnPlateau scheduler (factor 0.1, patience 5 epochs, cooldown 2 epochs, threshold 1e-4). Mini-batches randomly sample an in-context support set of 100 images in the first pretraining stage, and randomly sample between 30 and 500 in-context support images in the second context extension stage and the third finetuning stages. Each training stage runs for up to 100 epochs with early stopping based on validation loss (patience: 5 epochs). The training batch size is 80. We allocate 20% of the test set data for validation.

Computational cost. With an in-context support set of 100 images, our model predicts responses for ∼ 20 , 000 voxels in the higher visual cortex in under 20 seconds on a single RTX 6000 Ada GPU.

A.3 Text prompts for category-selective regions

We define a set of natural language prompts for each semantic category. For every image-prompt pair, we use the CLIP text encoder to generate text encodings. The natural language prompts for each category are listed below:

Faces [A photo of a person's face, A portrait photo of a face, A face facing the camera, A photo of a face, A photo of a human face, A photo of faces, People looking at the camera, A portrait of a person, A portrait photo]

Bodies [A photo of a torso, A photo of limbs, A photo of bodies, A photo of a person, A photo of people, A photo of a body, A person's arms, A person's legs, A photo of hands]

Places [A photo of a bedroom, A photo of an office, A photo of a hallway, A photo of a doorway, A photo of interior design, A photo of a building, A photo of a house, A photo of nature, A photo of a landscape]

Food [A photo of food, A photo of cuisine, A photo of fruit, A photo of foodstuffs, A photo of a meal, A photo of bread, A photo of rice, A photo of a snack, A photo of pastries]

Words [A photo of words, A photo of glyphs, A photo of a glyph, A photo of text, A photo of numbers, A photo of a letter, A photo of letters, A photo of writing, A photo of text on an object]

A.4 A more detailed description of logit scaling

The motivating factor underlying logit scaling is our desire to have our in-context learned encoder perform well regardless of the number of stimuli given to the model, and effectively generalize to context sizes beyond those seen during training. For example, while we may only train with between 30 and 500 images, a third-party experimenter may want to use fewer than 30 images or more than 500 images to condition the model. Across all cases, we want the model to succeed.

This logit scaling method was first proposed in [1, 2], and later adapted in Qwen LLM (logn-scaling) [3] and Llama 4 LLM (temperature scaling) [4]. We will briefly summarize the high-level math, which we take from [1] with our commentary:

Let the attention value in self-attention for a particular query token i to value token j to be

Substituting the expression for a i,j we have the entropy as

Then the entropy is defined as

we can express the formula as

If the first term is expressed as an expectation over j , we have

Which leads to the following approximation of entropy

If the vectors are assumed to be the output of a layernorm layer with length √ d , the expectation can be converted to one over the angles between vectors

Since most randomly distributed vectors in higher dimensions are orthogonal, we derive a term which can roughly be expressed as

where C does not depend on the number of tokens n .

This leads to an approximate scaling factor for the logits of log n to keep the entropy invariant to context length.

Therefore, we change the standard formulation of attention values by applying a scaling factor of log n to the QK T term, as shown in Equation (4) of our paper.

Note that in the above explanation we adopt the notation from [1].

A.5 Cortex prediction explained variance for different image encoding backbones and subjects

In this section, we compare three methods across multiple subjects (S1-S8) and embedding backbones (CLIP, DINO, SigLIP): the fully trained reference model fit to converge on each subject's full 9,000image training set; our BraInCoRL approach, which adapts to a new subject with only 100 in-context images; and a within-subject ridge regression baseline trained on the same 100 images with the BraInCoRL approach. In every case, BraInCoRL outperforms ridge regression and achieves accuracy similar to that of the fully trained model.

Fully Trained

BraInCoRL

Ridge Regression

0.0

0.7

Subj4

Subj2

Subj3

Subj1

Fully Trained

BraInCoRL

Ridge Regression

0.0

0.7

Subj8

Subj6

Subj7

Subj5

Fully Trained

BraInCoRL

Ridge Regression

0.0

0.7

Subj4

Subj2

Subj3

Subj1

Fully Trained

BraInCoRL

Ridge Regression

0.0

0.7

Subj8

Subj6

Subj7

Subj5

福

Fully Trained

BraInCoRL

Ridge Regression

0.0

0.7

Subj4

Subj2

Subj3

Subj1

Fully Trained

BraInCoRL

Ridge Regression

0.0

0.7

Subj8

Subj6

Subj7

Subj5

福

A.6 Voxelwise performance across five category-selective regions for different image encoding backbones and subjects

In this section, we report voxel-wise explained variance in five category-selective regions (faces, places, bodies, words, and food) with CLIP, DINO and SigLIP backbone for subjects S1-S8. We compare our in-context model (BraInCoRL) against the fully trained reference model fit to converge on each subject's full 9,000-image training set, within-subject ridge regression baselines trained on 100 and 300 images, and the FsAverage map. BraInCoRL outperforms the ridge baselines and closely approaches the performance of the fully trained model.

A.7 Voxelwise explained variance across varying support set sizes for more subjects, backbones and for pretrain-only models

In this section, we investigate how the size of the in-context support set affects voxelwise predictive performance. We evaluate three image-encoding backbones (CLIP, DINO, SigLIP) on eight subjects (S1-S8) by comparing the performance of BraInCoRL with the pretrain-only BraInCoRL (i.e. same architecture but only pretrained), the within-subject ridge-regression baseline, and the fully trained reference model fit to converge on each subject's full 9,000-image training set.

0.05

0.00

0.05

0.10

0.15

0.20

Subj 1

Explained Variance

CLIP

BraInCoRL

Pretrained

Fully Trained

0.05

0.00

0.05

0.10

0.15

0.20

DINO

BraInCoRL

Pretrained

Fully Trained

0.05

0.00

0.05

0.10

0.15

0.20

SIGLIP

BraInCoRL

Pretrained

Fully Trained

0.05

0.00

0.05

0.10

0.15

0.20

Subj 2

Explained Variance

BraInCoRL

Pretrained

Fully Trained

0.05

0.00

0.05

0.10

0.15

0.20

BraInCoRL

Pretrained

Fully Trained

0.05

0.00

0.05

0.10

0.15

0.20

BraInCoRL

Pretrained

Fully Trained

0.050

0.025

0.000

0.025

0.050

0.075

0.100

0.125

0.150

Subj 3

Explained Variance

BraInCoRL

Pretrained

Fully Trained

0.050

0.025

0.000

0.025

0.050

0.075

0.100

0.125

0.150

BraInCoRL

Pretrained

Fully Trained

0.050

0.025

0.000

0.025

0.050

0.075

0.100

0.125

0.150

BraInCoRL

Pretrained

Fully Trained

0

200

400

600

800

1000

Number of In-Context Examples

0.050

0.025

0.000

0.025

0.050

0.075

0.100

0.125

0.150

Subj 4

Explained Variance

BraInCoRL

Pretrained

Fully Trained

0

200

400

600

800

1000

Number of In-Context Examples

0.050

0.025

0.000

0.025

0.050

0.075

0.100

0.125

0.150

BraInCoRL

Pretrained

Fully Trained

0

200

400

600

800

1000

Number of In-Context Examples

0.050

0.025

0.000

0.025

0.050

0.075

0.100

0.125

0.150

BraInCoRL

Pretrained

Fully Trained

0.05

0.00

0.05

0.10

0.15

0.20

Subj 1

Explained Variance

CLIP

BraInCoRL

Pretrained

Fully Trained

0.05

0.00

0.05

0.10

0.15

0.20

DINO

BraInCoRL

Pretrained

Fully Trained

0.05

0.00

0.05

0.10

0.15

0.20

SIGLIP

BraInCoRL

Pretrained

Fully Trained

0.05

0.00

0.05

0.10

0.15

0.20

Subj 2

Explained Variance

BraInCoRL

Pretrained

Fully Trained

0.05

0.00

0.05

0.10

0.15

0.20

BraInCoRL

Pretrained

Fully Trained

0.05

0.00

0.05

0.10

0.15

0.20

BraInCoRL

Pretrained

Fully Trained

0.050

0.025

0.000

0.025

0.050

0.075

0.100

0.125

0.150

Subj 3

Explained Variance

BraInCoRL

Pretrained

Fully Trained

0.050

0.025

0.000

0.025

0.050

0.075

0.100

0.125

0.150

BraInCoRL

Pretrained

Fully Trained

0.050

0.025

0.000

0.025

0.050

0.075

0.100

0.125

0.150

BraInCoRL

Pretrained

Fully Trained

0

200

400

600

800

1000

Number of In-Context Examples

0.050

0.025

0.000

0.025

0.050

0.075

0.100

0.125

0.150

Subj 4

Explained Variance

BraInCoRL

Pretrained

Fully Trained

0

200

400

600

800

1000

Number of In-Context Examples

0.050

0.025

0.000

0.025

0.050

0.075

0.100

0.125

0.150

BraInCoRL

Pretrained

Fully Trained

0

200

400

600

800

1000

Number of In-Context Examples

0.050

0.025

0.000

0.025

0.050

0.075

0.100

0.125

0.150

BraInCoRL

Pretrained

Fully Trained

A.8 Impact of holding out the test subject's unique images during meta-training evaluated on more backbones

In this section, we further conduct ablations by evaluating a BraInCoRL model trained without holding out the test subject's support images ('BraInCoRL no HO') and a BraInCoRL model with only pretraining, on DINO and SigLIP backbones. The result indicates that fine-tuning on real neural data enhances performance, and that BraInCoRL is able to generalize effectively to entirely unseen images without having encountered them during training.

Subj1

Subj2

Subj5

Subj7

0.10

0.05

0.00

0.05

0.10

0.15

0.20

0.25

Pretrained

BraInCoRL

BraInCoRL no HO

Subj1

Subj2

Subj5

Subj7

0.10

0.05

0.00

0.05

0.10

0.15

0.20

0.25

Pretrained

BraInCoRL

BraInCoRL no HO

A.9 Correlation of each backbone's predictions with fully trained activation predictions

Using various image-encoding backbones, we plot how each subject's BraInCoRL predicted explained variance correlates with the fully trained model's explained variance (fully trained model refers to the fully trained reference model fit to converge on each subject's full 9,000-image training set). Across every backbone and all subjects, this correlation remains uniformly high.

Fully Trained

0.6

=X

=X

Subj1

0.4

0.2

0.0

Fully Trained

0.6

y=x

y=x

y=x

Subj2

0.4

0.2

0.0

Fully Trained

0.6

y=x

y=x

y=x

Subj3

0.4

0.2

0.0

Fully Trained

0.6

=X

=X

=×

Subj4

0.4

0.2

0.0

CLIP

DINO

SIGLIP

Fully Trained

0.6

y=x

y=x

y=x

Subj5

0.4

0.2

0.0

Fully Trained

0.6

y=x

y=x

y=x

Subj6

0.4

0.2

0.0

Fully Trained

0.6

=X

= X

=X

L!qns

0.4

0.2

0.0

Fully Trained

0.6

y=x

y=x

y=x

Subj8

0.4

0.2

0.0

CLIP

DINO

SIGLIP

A.10 Voxelwise explained-variance evaluation in BOLD5000 for more subjects and different backbones

In this section, we analyze how varying the number of in-context examples influences voxel-level prediction performance on the BOLD5000 dataset. For subject S2 and S3, we plot the mean Pearson's r between model-predicted and actual BOLD responses as a function of support-set size, comparing our BraInCoRL model against a ridge regression baseline. Results are averaged over five crossvalidation folds. Across all three image-encoding backbones (CLIP, DINO, and SIGLIP), BraInCoRL consistently outperforms ridge regression.

20

40

60

80

0.00

0.05

0.10

0.15

0.20

Pearson r

BraInCoRL

20

40

60

80

BraInCoRL

20

40

60

80

BraInCoRL

20

40

60

80

# In-Context Samples

CLIP

0.00

0.05

0.10

0.15

0.20

CSI3

Pearson r

BraInCoRL

20

40

60

80

# In-Context Samples

DINO

BraInCoRL

20

40

60

80

# In-Context Samples

SIGLIP

BraInCoRL

A.11 Dimensional reduction of predicted response function weights on more subjects

In this section, we utilize UMAP to visualize the BraInCoRL-predicted voxelwise weights under the CLIP image encoding backbone for subject S1-S8. The cortical maps show color-coded mappings that align well with functionally-defined regions.

Left

Hemisphere

Right Hemisphere

Subj1

Subj2

Subj3

Subj4

UMAP1

UMAP2

UMAP3

Anterior

Anterior

Superior

EBA

EBA

PPA

PPA

RSC

RSC

OPA

OPA

FFA-2

FFA-1

FFA-1

aTL-

faces

aTL-

faces

Food

region

Food

region

?

Left

Hemisphere

Right Hemisphere

Subj5

Subj6

Subj7

Subj8

UMAP1

UMAP2

UMAP3

Anterior

Anterior

Superior

EBA

EBA

PPA

PPA

RSC

RSC

OPA

OPA

FFA-2

FFA-1

FFA-1

aTL-

faces

aTL-

faces

Food

region

Food

region

FFA-2

A.12 Predicting cortical responses from natural language prompts on more subjects

In this section, we further predict cortical responses from natural language prompts on subject 2-8. For each semantic category, we convert a natural language prompt into a CLIP text embedding, project it into the image feature space, and use BraInCoRL to predict the corresponding voxel activation map. The predicted activations align closely with known t -statistic of category-selective region, illustrating the potential for zero-shot, language-driven functional mapping of visual cortex.

vmin

vmax

'A photo of a bedroom'

'A photo of words'

'A photo of a torso'

'A photo of a person's face'

probing

True

t

-statistics

vmin

vmax

'A photo of a bedroom'

'A photo of words'

'A photo of a torso'

'A photo of a person's face'

probing

True

t

-statistics

vmin

vmax

'A photo of a bedroom'

'A photo of words'

'A photo of a torso'

'A photo of a person's face'

probing

True

t

-statistics

vmin

vmax

'A photo of a bedroom'

'A photo of words'

'A photo of a torso'

'A photo of a person's face'

probing

True

t

-statistics

vmin

vmax

'A photo of a bedroom'

'A photo of words'

'A photo of a torso'

'A photo of a person's face'

probing

True

t

-statistics

vmin

vmax

'A photo of a bedroom'

'A photo of words'

'A photo of a torso'

'A photo of a person's face'

probing

True

t

-statistics

vmin

vmax

'A photo of a bedroom'

'A photo of words'

'A photo of a torso'

'A photo of a person's face'

probing

True

t

-statistics

A.13 Voxelwise prompt classification accuracy for more subjects

In this section, we further quantify the semantic specificity of BraInCoRL's voxelwise predictions on subject 3-8. we compute, for each subject and each category-selective ROI, the fraction of voxels whose peak predicted activation corresponded to the semantic category named by the text prompt.

A.14 Additional evaluation of BraInCoRL on NSD dataset

In this section, we provide two more evaluation metrics, namely Pearson R , and Spearman's rank correlation coefficient (Spearman's ρ ) for NSD dataset on Subject 1. In this case, the BraInCoRL model has not been trained or finetuned on Subject 1, while the Fully Trained model is trained on 9,000 images from this subject.

# In-Context Samples

Subj1

Pearson

r

# In-Context Samples

Subj1

Spearman's

ρ

A.15 Additional evaluation of BraInCoRL on BOLD5000 dataset

In this section, we provide two more evaluation metrics, namely explained variance and Spearman's rank correlation coefficient (Spearman's ρ ) for BOLD5000 dataset on Subject CSI1.

CSI1

# In-Context Samples

Expl. Variance

CSI1

# In-Context Samples

Spearman's

ρ

A.16 Evaluation of each training stage's contribution

In this section, we present the voxelwise explained variance for BraInCoRLtrained until different training stages. The results show that the progression from synthetic foundation → context flexibility → biological adaptation ensures that each fundamental challenge, namely response function coverage, variable context handling, and biological realism, is systematically addressed in the optimal order.

# In-Context Samples

Subj1

Expl. Variance

# In-Context Samples

Subj2

Expl. Variance

A.17 Performance of BraInCoRL conditioned on the full 9000-image set

In this section, we evaluate the performance of BraInCoRL with 9,000 in-context samples, for the four NSD subjects (S1, S2, S5, S7) focues by our main paper. The performance difference is less than 1% across all subjects when compared to a fully trained model (which is fit to converge on each subject's entire 9,000-image training set using gradient descent over multiple epochs). This means BraInCoRL achieves 94-99% of the fully trained model's performance.

A.18 Evaluation on the choice of loss function during training

In this section, we conducted an ablation study on the choice of different loss functions during the finetuning stage optimization and evaluated the model performance.

Our experimental results show that MSE and L1 losses achieve similar performance across all context sizes, with minimal differences. This suggests that both metrics are equally effective for capturing voxelwise neural response patterns.

In addition, the hybrid loss of 0 . 5 × MSE loss + 0 . 5 × (1 -cosine similarity ) underperforms by approximately 2-4% compared to MSE/L1. We argue this is because although cosine similarity captures directional relationships between predicted and true responses, this additional constraint limit the model's ability to accurately predict response magnitudes.

# In-Context Samples

Subj1

Expl. Variance

A.19 Ablation on logit scaling

In this section, we provide an additional ablation study to evaluate the effect of the logit-scaling, where we report the Voxel-wise explained variance of our BrainCoRL model and the model with the exact same structure but without logit scaling. It is shown that the logit scaling significantly boosts the model performance and generizability of various in-context support set sizes.

# In-Context Samples

Subj1

Expl. Variance

Appendix References

Jianlin Su. Analyzing the scale operation of attention from the perspective of entropy invariance, Dec 2021. URL https://kexue.fm/archives/8823 .

David Chiang and Peter Cholak. Overcoming a theoretical limitation of self-attention. arXiv preprint arXiv:2202.12172 , 2022.

Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXiv:2309.16609 , 2023.

Meta AI. The llama 4 herd: The beginning of a new era of natively multimodal ai innovation. 4(7):2025, 2025. URL https://ai.meta.com/blog/ llama-4-multimodal-intelligence/,checkedon .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose BraInCoRL as a model for predicting neuronal fMRI responses. The approach uses a transformer-based model with meta-learning across voxels and subjects, and few-shot in-context learning across stimuli without the need for finetuning. The authors show that the model generalizes across subjects, stimuli and recording acquisition parameters, outperforming anatomical and subject-wise baselines. They also explore attention patterns and carry out ablations as well as link visual cortex activity to natural language.\n\nSTRENGTHS AND WEAKNESSES:\nOverall, this study presents a solid modelling framework for fMRI responses. The methods are well evaluated and different applications are demonstrated. Minor weaknesses include lack of justifications of design decisions. My questions further elaborate on this.\n\nQuality:\nThe general problem is formalized cleanly. The authors perform a rigorous evaluation, making proper use of held-out data, and performing ablations. Baseline models are well selected and demonstrate the benefits of the proposed model and training, where no finetuning is required.\n\nClarity:\nThe paper is very well written. Figures and illustrations are clean and informative. The appendix also provides extensive methodological details on the proposed model and training. Minor points:\nEq. 4: redefines $c$ to represent context length. It might be better to use another symbol here to avoid confusion. Figure 7 should label the y-axis.\n\nSignificance:\nThe authors formulate the encoding problem as a meta-learning task, which addresses the problem that previous encoders were subject specific, requiring large individual datasets. The proposed model and training techniques demonstrate the generalization capabilities of modern models and training. Similar techniques might be applicable to other recording modalities or domains. The main immediate beneficiaries are practitioners using fMRI encoding models.\n\nOriginality:\nThere are previous studies on multi-subject approaches, as cited by the paper [30, 32], which do require finetuning. So the main advance of this work is the complete absence of needing finetuning. The authors adopt many techniques that were developed for LLMs, such as analysis-by-synthesis and making use of logit scaling for variable context length. These are adapted without proposing entirely new techniques but are novel in the context of fMRI encoders. Contrary to previous approaches, each voxel has a response function that is addressed in a meta-learning task. A transformer generates parameters of the visual response functions based on context tokens.\n\nQUESTIONS:\n1. The authors assume that the unobserved true response function is noiseless, i.e. all the noise is part of the measurement process. In reality there will be different noise components, some of which will be inherent to the brain response. Could it be useful to model these different sources in more detail?\n\n2. The visual response function is designed and required to be differentiable. How much of a limitation is this?\n\n3. Line 138: The authors used MSE optimization. Did they consider other error metrics?\n\nLIMITATIONS:\nLimitations of the study are discussed in a dedicated section and mention static vs. dynamic stimuli, as well as dataset diversity. Societal impacts are discussed in an appendix section. In my opinion, these sections address all reasonable points.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-29"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work proposes a meta-learning approach to modeling fMRI recordings in response to visual stimuli. The method allows for better generalization between subjects and even recording conditions, taking a step beyond subject-specific encoding models. A transformer is trained to predict linear-readout weights given a relatively small set of in-context image-feature, voxel-activation pairs. The model is trained on the NSD dataset and tested on held-out subjects, as well as the BOLD5000 dataset. Results show that the proposed method explains more variance than direct optimization using Ridge regression across a variety of in-context training set sizes. Further, the manuscript presents clusterings of predicted predicted readout weights in accordance with known category-selective brain regions. An analysis of the images with highest attention scores in the last transformer layer reveals that images tend to stem from the category preferred by the respective brain region.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n\n1. The idea of applying meta-learning to fMRI data is interesting and as noted in the manuscript, it is a promising approach towards more subject-general models.\n\n2. The results clearly show data-efficiency and transfer abilities of the proposed model. This has strong relevance for the field.\n\n3. Experiments are quite rigorous, testing multiple models and datasets.\n\n4. Cluster analyses and semantic characterizations of the predicted voxel weights show clear correspondences to established category-selective regions, showing that the model does learn meaningful features.\n\n5. The manuscript is clear and easy to follow for the most part.\n\n**Weaknesses**\n\n1. Fig. 3b) shows that the performance difference between the pretrained model and BrainCoRL is non-zero, but relatively small. A lot of the advantage over Ridge regression is seemingly not attributable to explicit leveraging of the structure of neural data.\n\n2. The variance explained by all models is relatively low, which means the model is competitive, but at a low overall level. We observe that it is able to capture very rough trends like category selectivity, but it is difficult to gauge whether learning more fine-grained features driving single voxels is possible with this method. Since a voxel's tuning properties are inferred by studying the tuning properties of other voxels, I could imagine a smoothing effect where voxels with very distinct tuning properties are not modeled accurately. Therefore, while the feature analyses in Fig. 6-8 are interesting, they are not fully convincing for me. This is a drawback but not a strong criticism for this work – I acknowledge that this can be left for future efforts.\n\nQUESTIONS:\n1. A brief overview of the model architecture would be good to include in the main text.\n\n2. In l.131 should it be [x_i;\\beta_i]? Or what does v_i denote exactly?\n\n3. Equation (4) came very out of the blue for me – perhaps this could be explained more thoroughly\n\n4. Is it possible to use the full 9,000 images as in-context samples, thereby actually reaching the performance of the fully trained model?\n\n5. The results reported on page 6 are somewhat unclear to me. It says '...we evaluated BrainCoRL with each subject's 9000 unique images as in-context support set...'. Then in the next sentence '...using just 100 test(?) images', In the table caption, it says '...uses just 100 in-context images...'. Perhaps this could be clarified a little.\n\n6. In Fig.4 for BOLD5000, why are so few samples used compared to NSD?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 4, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces BraInCoRL, a novel transformer-based model for brain encoding that performs meta-learning and in-context learning to predict voxel-level responses in the human higher visual cortex to visual stimuli. Unlike traditional brain encoders that require subject-specific large-scale data, BraInCoRL is trained across subjects and generalizes to novel subjects and datasets using only a small number of in-context examples (i.e., image–response pairs).\n\nSTRENGTHS AND WEAKNESSES:\nStrong Generalization Across Subjects and Datasets\nBraInCoRL shows consistent improvements across novel subjects and even across datasets collected with different scanners and protocols (e.g., NSD to BOLD5000). It operates effectively with just a few in-context images, outperforming ridge regression baselines and closely matching fully trained models using 9,000 images.\n\nModular and Versatile Model Design\nThe model is constructed modularly with a frozen CLIP feature extractor and a higher visual cortex transformer that can generate hyperweights for voxel-wise encoding. This enables efficient processing of both visual and language inputs, facilitating downstream tasks such as natural language–driven cortical querying.\n\nInsightful Analyses on Model Behavior\nThe paper presents compelling insights into the behavior of the model:\nIt learns to attend to semantically relevant images when predicting voxel responses in category-selective regions.\n\nSurprisingly, it performs better when in-context support sets are randomly sampled rather than overly specific, suggesting that semantic diversity enhances generalization—a notable finding for the design of few-shot neural systems.\n\nQUESTIONS:\nComplexity and Scalability of the Training Pipeline\n\nThe proposed three-stage training strategy—synthetic pretraining, context extension, and real-data finetuning—is well-motivated and effective. However, it introduces significant complexity and resource requirements. In practical settings, especially those with limited access to large-scale fMRI datasets or compute, this may hinder broader adoption. Additionally, the contribution of each stage to the final model performance is not thoroughly disentangled.\n\nSuggestion for Authors\nIt would strengthen the paper to include a quantitative evaluation of each stage’s contribution to performance. For instance, how much does synthetic pretraining alone contribute to generalization? How critical is the context extension step? Ablations isolating each component, or alternative streamlined training setups, would help clarify the necessity of the full pipeline and guide future, potentially more efficient implementations.\n\nOverall comments:\n\nThis paper presents a compelling advance in the intersection of computational neuroscience and meta-learning. The proposed BraInCoRL model is technically innovative, empirically strong, and provides new insights into how generalizable brain encoding models can be built.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper addresses an important problem: how to construct cross-subject, few-shot fMRI encoding models. From my understanding, the paper appears to use a small set of image- (single) voxel pairs as a support set, and based on this, trains a model to output the parameters of voxel-wise encoding model. However, I did not fully grasp some details of the proposed method. The paper evaluates its approach on datasets such as NSD, and the experimental results show that the proposed method can effectively encode fMRI signals.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n+ The paper addresses an important research problem and achieves promising results.\n+ The paper provides a relatively comprehensive review of related work.\n+ The meta-learning method proposed in the paper is thought-provoking and offers valuable insights.\n\nWeaknesses (Details refer to Questions):\n+ In my opinion, the presentation of the method should be more detailed to improve clarity and comprehensibility.\n\nQUESTIONS:\nFor the methodology section, I have the following unclear points:\n+ In line 131, the context tokens c is indexed by i, which seems to imply that its range is from 1 to p, however, in line 132, its range is from 1 to k. it seems inconsistent?\n+ The authors use beta to represent the voxel's BOLD signal (such as in line 129), but in the context tokens c, they directly use the \"voxel\" v. Does this v here refer to a vector similar to a positional encoding?\n+ The authors employ the technique of logit scaling in Section 3.3; however, I am not familiar with it. Could the authors provide a clearer explanation to clarify its underlying principle?\n\nFor the experimental section, I have the following questions:\n+ What is explained variance? Why is only this metric used in the evaluation on the NSD dataset, while the Pearson correlation coefficient is not assessed?\n+ Are there additional metrics to directly assess the accuracy of fMRI encoding?\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}