LaVCa: LLM-assisted Visual Cortex Captioning
Authors: Takuya Matsuyama
Shinji Nishimoto
Yu Takagi
Venue: ICLR 2026 Poster (2026)
Keywords: Neuroscience
Computer vision
Visual systems
Captioning
Large language model
Semantics
Neuroimaging
Functional magnetic resonance imaging
Primary area: applications to neuroscience & cognitive science
TL;DR: We propose LaVCa, a novel method that generates data-driven captions for individual voxels.

Abstract: Understanding the properties of neural populations (or voxels) in the human brain can advance our comprehension of human perceptual and cognitive processing capabilities and contribute to developing brain-inspired computer models. Recent encoding models using deep neural networks (DNNs) have successfully predicted voxel-wise activity. However, interpreting the properties that explain voxel responses remains challenging because of the black-box nature of DNNs. As a solution, we propose LLM-assisted Visual Cortex Captioning (LaVCa), a data-driven approach that leverages large language models (LLMs) to generate natural-language captions for images to which voxels are selective. By applying LaVCa for image-evoked brain activity, we demonstrate that LaVCa generates captions that describe voxel selectivity more accurately than the previous approaches. The captions generated by LaVCa quantitatively capture more detailed properties than the existing method at both the inter-voxel and intra-voxel levels. Furthermore, we find richer representational content within cortical regions that prior neuroimaging studies have deemed selective for simpler categories. These findings offer profound insights into human visual representations by assigning detailed captions throughout the visual cortex while highlighting the potential of LLM-based methods in understanding brain representations.

Published as a conference paper at ICLR 2026
LAVCA: LLM- VISUAL CORTEX CAPTION- ING
Takuya Matsuyama 11221177@ecs.osaaauu jp
Shinji Nishimoto 1.2+
nishimoto. shinji fba@osaka-u -u ac. jp
Yu Takagi 1,,3+
1,33+
takagi auenitech.ac..pp
Uniivesiity Osaka, Japan
National Institute of Information and Communications Technology, Japan 3 Nagoya Institute of Technology, Japan
Sümull
Captioning
Voxel
imaga
Measured vexel activation
Cwodth (henaa)
Voxel
Interpret
A man with a loothbrush
street sign on the sido
LavCa' (Ours)
Acc 0.127
Brain
innovative entrance overlape with a train bridge.
charachar.
person and cartoon
Acc.
0.270
LaVCa
name Ian Charrpbeeeactcs angu- (tilled abark.
Figure 1: Illustration of our paper. Our proposed method, LaVCa, produces natural language captions that provide a fine-grained description voxel selectivity (representation) and more accurately cap- ture the characteristics of brain activity in the visual cortex, outperforming conventional approaches.
ABSTRACT
Understanding the properties of neural populations (or voxels) in the human brain can advance our comprehension of human perceptual and cognitive processing capabilities and contribute to developing brain-inspired computer models. Recent encoding models using deep neural networks (DNNs) have successfully predicted voxel-wise activity. However, interpreting the properties that explain voxel sponses remains challenging because of the black-box nature of DNNs. As a solution, propose M-aassiidee Visual Cortex Captioning (LaVCaa, data- driven approach that leverages large language models (LLMs) to generate natural- language captions for images which voxels selective. By applying LaVCa for image-evoked brain activity, we demonstrate that LaVCa generates captions that describe voxel selectivity more accurately than the previous approaches. The captions generated by LaVCa quantitatively capture more detailed properties than the existing method at both the inter-voxel and intra-voxel levels. Furthermore, we find richer representational content within cortical regions that prior neuroimaging studies have deemed selective for simpler categories. These findings offer profound insights into human visual representations by assigning detailed captions through- out the visual cortex while highlighting the potential of M-L.-based methods understanding brain representations.
"quual last author.

---

Published as a conference paper at ICLR 2026
INTRODUCTION
primary goal of computer vision is build systems capable of processing and understanding the complex visual world in a manner akin to human perception. Studying how the human brain-witt its advanced visual functions- forms its visual representations deepens our understanding of the brain's visual network and holds promise for developing next-generation computer vision models.
Over the past decade, encoding models have become the standard tool for this endeavour (Kay et al. 2008 Nishimoto al. [211]] Naselaris et al. (01I)) Early work employed handcrafted, low-level filters or one-hot semantic labels. yielding interpretable- but coarse descriptions of voxel-level (the spatial measurement unit of fMRI) selectivity. Modern approaches substitute deep neural- -netrokk (DNN) features, which dramatically raise prediction accuracy (Güçlü & Van Gerven [015 Scchrmppf et al. 2021 Takagi & Nishimoto 2023). Yet the very richness that makes DNNs powerful also renders them opaque: it remains difficult to explain why given voxel activates, especially at the single-voxel level where group-averaged semantic axes (Huth et al. 2016 Lescroart Gallant 2019)
are too blunt.
In this study, we address the difficulty of voxel-level interpretation with new method called LLM- assisted Visual Cortex Captioning (LaVCa), which generates data-driven captions for individual voxels (Figure J. La proceeds four steps: (1) building voxel- wiie encoding models for brain activity evoked by images, (2) identifying the optimal images for each voxel's 's encoding model using an augmented image dataset, (3) generating captions for these optimal images, and (4) creating concise summaries from those captions. By leveraging large language models (LLMs) with access to a vast, open-ended vocabulary, LaVCa generates diverse inter-voxel captions. Moreover, generating captions from multiple keywords enables to capture diverse intra-voxel properties.
Our contributions are as follows:
1. We propose LaVCa, which leverages LLMs to generate natural language captions of voxel-level visual selectivity. By adopting a multi-stage design that decomposes the captioning process into interpretable steps, LaVCa enhances interpretability compared prior work while preserving deseriptive richness.
Prediction accuracy (sentence-level
0.25
50
$-100
40
50 100
$10
0.24
45
0.23
0.22
2. We demonstrate that LaVCa produces more accurate captions than the earlier method
BrainSCUBA (Luo al.) (023) and better characterizes voxel-wise visual selectivity
0.21
+ LaVCa (Ours)
(Ours)
BrainSCUBA
through brain activity prediction.
0.20
10a
102
10*
3. We also demonstrate that LaVCa can gen- erate highly interpretable and accurate cap- tions without sacrificing information from the optimal images (Figure 2
Number of words (log-scale)
Figure 2: The relationship between brain ac- tivity prediction accuracy and voxel caption length (number of words) for single subject (subj01). Numbers next to each point denote the number of optimal image captions em- ployed by La for summarization and by Concat for direct concatenation.
4. The captions generated by LaVCa quantita- tively capture more detailed properties than the existing method at both the inter-voxel and voxel levels.
5. More detailed analysis of the voxel- specific properties generated by LaVCare- veals richer representational content within ROIS that prior neuroimaging studies have deemed selective for simpler categories.
2 RELATED WORK
Two complementary approaches frame modern fMRI research: encoding and decoding models Naselaris et al. 2011). Encoding models aggregate activity across many stimuli for each voxel to
2

---

very Published as a conference paper at ICLR 2026
Vinsal stimul
Brain response
External image dilasse
(MILA
Corresponding cuptions
Contrectiv
Contreetive
ancoding weight
tangua
VLM
Bllering
Voval keywords
[caption]
"aeiie catrod 'graddands"" rkanling chile"
"Wide werer.
The wethin Teaeuu fucty be more und lmnts.
Keysorda (LUM)
Sentince
"eternctin betweer
Trotnebbee
husnann and andinde
Figure 3: Architecture of LaVCa. We construct a voxel-wise encoding model for human subject's brain activity data (measured using (MRI) while viewing images, using latent representations from contrastive vision- language model (VLM). The encoding weight is obtained through ridge regression. b We identify the optimal images for given voxel by calculating the inner product between the contrastive VI.M embeddings of external image datasets and the voxel's trained encoding weight, selecting the top-N images (the "optimal image set") that produce the highest predicted activation. с Next, we use a multimodal large language model (MLLM) to generate captions for each optimal image set, allowing an LLM to interpret them. Finally, we prompt LLM to extract keywords from the captions, filter these keywords, and feed them into a "Sentence Composer," producing concise voxel caption.
pinpoint the features that best explain its responses. Decoding models reverse the mapping, pooling activity across many voxels to reconstruct participant's moment-to-mee to-moment percepts percepts-ranging from continuous speech (Tang et al. 2023) to images (Takagi & Nishimoto) (023)) Relatedly, recent brain- to-text decoding studies align brain activity with models to generate natural- lanural-language descriptions directly from neural signals (Chen et al. 2025a b). Although both lines of work exploit powerful deep models, they address distinct questions: decoders ask "What was the observer perceiving?? whereas encoders ask "What information does this voxel represent?." The present study targets the encoding side, judging captions by how accurately they predict voxel responses rather than how faithfully they reproduce the original stimulus.
Early encoding work used handcrafted low-level filters or one-hot semantic labels, enabling straight- forward but coarse voxel interpretation (Kay al.) 2008 Nishimoto et al.) 2011 Naselaris et al.)
2011; Huth et al.] 2012). Swapping these features for deep-neural- -ntworkk (DNN) embeddings
dramatically improves prediction accuracy (Güççīä & Van Gerven 2015 Schrimpf al. 2021 Takagi & Nishimoto 2023 Antonello et al. 2024), yet the high-dimensional representations make individual voxels hard explain. Population- level remedies project many voxels onto few semantic axes
(Huth et al. 2016 Lescroart & Gallant 2019 Nakagi et al. 2024). but sacrifice single-voxel nuance.
To obtain finer, voxel-specific explanations, data-driven eexteeeneattion approaches such Brain- SCUBA (Luo et al.) (2023) and SASC (Singh et al. 2023) have been proposed. BrainSCUBA is an end-to-end method that uses an existing image captioning model produce voxel-wise captions for the visual cortex, whereas SASC uses an LIM merge multiple short phrases-thor with the highest predicted voxel activations into a single, data-driven caption, thus describing the semantic properties of voxels. However, their reliance single captioning model (BrainSCUBA) short n-gram phrases (SASC) limits lexical richness and adaptability.
By (i) decoupling image selection from caption generation, (ii) using LLM-based keyword extraction followed by lightweight sentence composition (iii) allowing any vision- languaee backbone, and (iv) working with any LLM that has strong language skills without tpsk-specific retains the high predictive power of brain activity while yielding richer and more controllable voxel-level descriptions than prior work.
LaVCa
3

---

Published as a conference paper at ICLR 2026
3 METHODS
3.1 FMRI DATASET
This study uses the Natural Scenes Dataset (NSD) (Allen et al. 2022) following the same experimental conditions as in BrainSCUBA. The NSD consists of data collected over 30 to 40 sessions using Tesla IMRI scanner, with each participant viewing 10,000 images, repeated three times. We analyze data from the four participants (Subject 01, Subject 02, Subject 05, and Subject 07) who completed all imaging sessions. The images and captions used in NSD are drawn from MS COCO and resized to 224 x 224 pixels to align with the input requirements of the vision models used. We average the brain activity data for each subject across repeated trials of the same image to improve the signal-to-noise ratio. Up 9,000 images per subject used as training data, and the remaining 1,000 images reserved for testing, We use the preprocessed scans with a resolution of 1.8 mm provided by NSD for the functional data. We use single-trial beta weights estimated via generalized linear model within ROIs. Moreover, we standardize the response of each voxel to have mean of zero and a variance of one within each session. We use the ROIs provided by NSD, which include early and higher- (ventral) visual areas and face, place, body, and word-selectin regions.
3.2 LLM-ASSISTED VISUAL CORTEX CAPTIONING (LAVCA)
We propose method, LaVCa (Lss-assisted Visual Cortex Captioning), to automatically generate data-driven natural language captions that characterize each voxel's selectivity the visual cortex. La consists of four stages (Figure 3):
1. Construct voxel-wise encoding models for each subject while they view natural images.
2. Identify the optimal image set by finding the top-N images that most strongly activate each voxel (according to the trained encoding models).
3. Generate captions for these optimal images using a multimodal large language model (MLLM) for summarization by LLM in the next step.
4. Derive concise voxel captions by extracting and filtering keywords from the image captions, then feeding these keywords into a "Sentence Composer."
We describe the core pipeline here; ablations are detailed in Appendix A.3
3.2. ENCODING MODEL CONSTRUCTION
First, we construct voxel-wise encoding models to predict each voxel's activity in response to natural images (Figure 3). To obtain high-level feature representations of visual stimuli that can be linked to neural responses, we use embeddings from a contrastive vision-language model (VLM; e.g.. CLIP (Radford et al. ))2T)). Specifically, for comparability with BrainSCUBA, we adopt the projection layer embedding of CLIP's vision branch and use the same pretrained checkpoint as We also re- re-implemented BrainSCUBA in in-house; note that our implementation differs in the dataset used for the projection step and in the training approach for the encoding model (Appendix A.2.4 for details).
reported in that work (see Appendix(A.22
For each image stimulus i, we extract its CLIPP- projection-layer embedding x; E IRd, L.2- normalize unit norm, and pair with the measured responses across all voxels y, E" The encoding model assumes a linear relationship
x; +
where W e Rexd represents the voxel-wiss encoding parameters and e; captures residual noise. We estimate W using ridge regression the NSD training set.
3.2.2 EXPLORATION OF OPTIMAL IMAGE SETS FOR VOXELS
Next, identify the optimal image set for each voxel (Figure3p) We compute the inner product between the voxel's encoding weight and CLIP- Vision latent representations from a large-scale external dataset (distinct from NSD) obtain predicted voxel responses for each image. We then
4

---

Published as a conference paper at ICLR 2026
select the top-N images that generate the highest predicted activation. This process is equivalent to calculating the predicted responses of each voxel for every image. This study uses approximately 1.7 million images from Openlmages- -6 (Kuznetsova et al. 2020)
3.2.3 CAPTIONING OPTIMAL IMAGE SETS WITH MLLM
To enable LLM to interpret each voxel's optimal image set, we first generate captions for these image sets using an MLLM. We use MiniCPM-V Yao et al. 2024) with the prompt "Describe the image briefly." " For our accuracy evaluation, we also form a simple baseline by concatenating the top-N captions from the optimal image set.
3.2.4 GENERATING VOXEL CAPTIONS
Finally, we generate interpretable voxel captions from the image captions. First, an LLM to extract common keywords across the captions within each voxel's optimal image set (Figure 3d). Following the in-context learning prompt approach from (Dunlap al. 2024), extract multiple key- words from the caption sets using an LLM (A6 for the prompt). We use gpt-4o (gpt-4o-2024-08- in the OpenAI API) as the LLM. To remove irrelevant or noisy keywords, similarity between each keyword's embedding from CLIP" text branch (prompted as "A photo of (keyword).") and the encoding weight for that voxel, then apply a softmax threshold retain only sufficiently relevant keywords. Hereafter, we refer to CLIP's text branch as "CLIP-Tess Text", Next, we transform these filtered keywords into a sentence-level caption using the "Sentence Composer" from MeaCap (Zeng et al., 2024). initially designed to generate image captions from keyword sets. MeaCap can generate a caption by inputting the target image's keywords into the Sentence Composer while referencing similarities to the image features. In this study, we replace image features with encoding weights so that the model composes a coherent sentence from the voxel-specific keywords (for details, see Section A.2.I)
compute the cosine
3.3
CAPTION EVALUATION
3.3. 1
BRAIN ACTIVITY PREDICTION AT SENTENCE LEVEL
voxel caption that truly reflects a voxel's selectivity should be more similar the caption of an NSD image that strongly activates that voxel, and less similar to captions of images that do not. We therefore predict voxel-wise brain activity from sentence similarity evaluate how accurately each caption captures voxel selectivity (Figure A7p) Importantly, this procedure differs conceptually from decoding, which predicts caption for every stimulus. Following (Singh al. 2023), we:
1. Use a pretrained Sentence-BERT compute text embeddings for each voxel caption and each NSD image caption.
2. Compute the cosine similarity between the voxel caption embedding and each NSD image caption embedding.
3. Treat this similarity value as the predicted activity for that voxel on that image.
For each voxel v, we then calculate the Spearman's 's rank correlation between the vector of predicted sentence-level similarities and the measured activity; this correlation coefficient is regarded as the prediction accuracy for that voxel.
For statistical significance. we use a permutation test to assess voxel-wise prediction accuracy. Multiple comparisons are corrected using the Benjamini- Senjamini-Hochberg false discovery rate procedure (a = 0.05). Detailed procedures are provided in Appendix A.2.2 A.2..
3.3.2 BRAIN ACTIVITY PREDICTION AT IMAGE LEVEL
Because sentence-based evaluation can be influenced by non-visual linguistic features (e.g., sentence length, clarity of phrasing, or stylistic variation), we also assess voxel selectivity using image similarity (Figure A7p). We use FLLUX.1-schnd to create voxel image and then compute vision embeddings (via CLIP-Vision for both the generated voxel image and each NSD trial image. We obtain an image-level metric of predicted brain activity by comparing these embeddings, focusing
5

---

Published as a conference paper at ICLR 2026
Prediction accuracy (el
b
Prediction
LaVCa (Durs)
DFA
Figure 4: Mapping of brain activity prediction accuracy (subj01). a The sentence-level prediction performance is projected onto inflated cortical surfaces (top: lateral, medial, and dorsal views) and flattened cortical surfaces (bottom, with the occipital areas at the center) for both hemispheres. Voxels with significant prediction performance are color-coded (all colored voxels P < 0.05, FDR corrected). The white outlines indicate the ROIs that are among the top two in terms of the total voxel count across subjects for each semantic category erdddory-ddyy (Extra Striate Body Area; EBA, and Fusiform Body Area; FBA-2), Face (Fusiform Face Area; FFA-1, and Occipital Face Area; OFA), and Places (Parahippocampal Place Area; PPA, and Occipital Place Area; OPA). Word areas are shown in Figure A8 b A comparison of sentence-level prediction performance between our method, LaVCa, and the existing method, BrainSCUBA on the flattened cortical surface.
purely on visual content. Crucially, this procedure is not image reconstruction in the decoding sense; it characterises voxel selectivity through images rather than attempting to recreate the stimuli themselves.
RESULTS
4.1
VOXEL ACTIVITY PREDICTION
We examine whether LaVCa can generate concise and interpretable voxel captions without losing critical information in each voxel's optimal image set. We compare two approaches from the perspective of interpretability by varying the number of optimal images used by LaVCa (Top-N) and by simply concatenating the captions of the optimal images (Concat-NV). Figure plots prediction accuracy against the average caption length on the horizontal axis, highlighting the trade-off between accuracy and interpretability. Concat-N achieves better accuracy N increases (up to N 10) but at the cost of a much longer caption, which can reduce interpretability. In contrast, LaVCa merges information across the optimal image set into concise summary, retaining interpretability even N grows and reaching accuracy comparable to Concat-N (see Figure A10b). Results for all participants are provided Figure A10
Next, we determine whether the generated captions accurately capture the properties of voxels in the visual cortex. To this end, we map sentence-level prediction accuracy onto both inflated and flattened cortical surfaces (Figure These maps illustrate that LaVCa captions significantly predict voxel activity throughout the visual cortex (P 0.05, correcedd).. Results for all subjects the sentence and image levels are presented in Figures A8 and A9
Finally, we compare two configurations of La its default five-keyword version with the Sentence Composer and simplified single-keyword variant without the Sentence Composer- against the existing method BrainSCUBA and a shuffled variant (LaVCa captions shuffled across voxels) at both the sentence and image levels, focusing the top 5,000 voxels with the highest accuracy on the training data (Table D) Our proposed method, LaVCa, outperforms BrainSCUBA and the single- keyword variant (P 0.05, paired t-test). This finding suggests that using multiple keywords and composing them into a coherent sentence provides a more accurate explanation of voxel selectivity. Importantly, LaVCa outperforms BrainSCUBA at the image-level, indicating that the improvement

---

Published as a conference paper at ICLR 2026
Table I: Comparison of brain activity prediction accuracy at the sentence and image levels. For each subject, the mean and standard deviation of accuracy on the test data are displayed for the top 5,000 voxels with the highest accuracy on the train data.
Sentence level
Model
# keywards
Sentence Composer
subj01
subj02
subjUS
subjU?
Shutfled
0.007 199
0.058 +: 0.223
0.068 + 0.243
0.009 0.175
BrainSCUBA
0.207 a 0.062
0.251 + 0.071
0.264 a 0.084
0.182 a: 0.065
LaVCa (Ours)
0.205+0.066 068
0.250+0.075
0.27240.086
0.186+0.072
LaVCa (Ours)
0.246 0.066
0.287 = 0.075
0.306 + 0.084
0.218 = 0,073
Image level
Model
keywords
Sentence Composer
subj01
subj02
subj0S
subj07
Shuflled
0.163
+:
0.0666
0.009 0.149
BrainSCUBA
0.188 0.067
0.226 + 0.070
0.250 + 0.078
0.169 + 0.069
LaVCa (Ours)
X
0.182 0.063
0.221 : 0.066
0.252 :: 0.077
0.158 0.064
LaVCa (Ours)
0.072
0.070
0.187
not merely due better handling of non-visual linguistic features (e.g., sentence length or phrasing), but reflects a genuinely enhanced characterization of visual selectivity. Furthermore, Va achieves far higher accuracy than the shuffled condition. Results for the top 1,000, 3,000, and 10,000 voxels appear in Table A8 and A9 After visualizing sentence-level prediction accuracy across the cortex, we find that La VaV exceeds BrainSCUBA's performance throughout the visual cortex (Figure 4b).
See Figure A8 for the results of all subjects.
4.2 LEXICAL AND SEMANTIC DIVERSITY ANALYSIS
We next assess how effectively LaVCa captions capture both lexical and semantic diversity across voxels, focusing first on inter-voxel diversity (Table left). For this quantitative evaluation, we use three metrics: (1) the total vocabulary size (excluding stop-worde across all voxel captions (Lexical); (2) the average variance across each dimension of the CLIP-Text embedding computed on all voxel captions (Semantic); and (3) the number of principal components (PCs) required to capture 90% of the variance of CLIP--Txt embedding across captions in a principal component analysis (PCA; Semantic).
First, we evaluate the diversity of LaVCa captions compared with the existing method, BrainSCUBA. When averaged across subjects, LaVCa markedly outperforms BrainSCUBA in both lexical (16,922 vs. 3,193 in vocab. size) and semantic (0.0642 vs. 0.0588 in variance of embeddings; 219 vs. 127 in PCs required for 90% variance explained) diversity. These findings confirm that our open-ended LLM-based approach can produce richer word usage and more meaningful captions across inter-voxel comparisons.
We evaluate the diversity of L. VCa captions compared with more detailed captions. BrainSCUBA leverages ClipCap (Mokady et al., 2021), model that produces relatively simple image captions. We use the top-1 captions generated by the MLLM on the optimal image sets (equivalent to the case where N = in Concat-N) to compare the diversity of LaVCa with more detailed captions. When averaged across subjects, Top-1 (13,959 vocab. size, 0.0638 avg. variance, 210 PCs) exhibits both vocabulary range and semantic diversity close to LaVCa. However, La achieves a higher prediction accuracy (0.264 vs. 0.224), indicating that LaVCa can preserve robust brain activity prediction performance while enhancing the diversity of generated captions.
Next, we evaluate diversity from an intra-voxel perspective by comparing captions generated by three models in both lexical and semantic dimensions (Table A10 right). We use three metrics: (1) the vocabulary size of each voxel's caption (Lexical), (2) the average sentence length in each voxel's caption (Lexical), and (3) the average variance across all dimensions of Word2 embeddings of each caption's 's words (excluding stop-words) (Semantic). When averaged across subjects, LaVCa markedly outperforms BrainSCUBA in both lexical 111.4 vs. 6.09 in vocab. size) and semantic (11.9 vs. 6.19 in avg. length; 0.0199 vs. 0.0160 in variance of semantic embeddings) diversity. This improvement suggests that LaVCa more precisely captures the fine-grained intra-voxel characteristics.
7

---

Published as a conference paper at ICLR 2026
a
EBA
OFA
PPA
OWFA
Derson
SPA SPAnsconst ruc
bu
ding
tion
ttem
per
noun
Betivity
sport
per
S
SON
COLL ittmm dessen
FOOD
Kitchen
SIER
ODTeCT
1000
CCLLO AIR
TOOm
screet
JWEN
DUDS
decorate
fAce ANDEO
S show
seang
Show COI
face
verb
Faay
show
PaInT
POSS Le
Show.weat
S
Smil
hold
Take
O over PrLok
se
ohe
decorate
hold aaloos
decoraaee
heme
aop wer Inclute Mineer YOUNd!
Include eature
b
"Ie Reddeed cebeare
leny Sae paber one
"Thee deceested every
oegas.
Peahod
by the Smardt
riatag OF mib 213 dack
perticalete calerned
homoreas detall loed)
taage
deemed dosert bry decerated Bis peand calos
wilh
"The pensee backiin the slgen peirs
wlole bewitking moneatiiiig
freeting Day repraiio"
whiltk cerking for
sporting
vegeeible
"Eackciatiy thened mas fastano themad the lucs, with a docccand antiaal citnerriiioo iis Sumardd dumscrt""
popll rtancing diyple tack lmagin gracy eepreveninng teiiinng reakk setting""
prest maads Foed Doom daglayed calerttd akjust.
Figure 5: Interpretation of LaVCa captions (subj02). a UMAP projection of caption text across four ROIs (EBA, OFA, PPA, OWFA), visualized on flatmap (top). word cloud the 100 most frequent nouns in these captions (middle), colored by location in the UMAP space. A word cloud of the 100 most frequent verbs (bottom). Visualization of the top two captions (by accuracy) for eight clusters on the flatmap in OFA. The images generated for each caption appear to the left or above the text. Voxels are connected to their corresponding captions and images by lines. The color of each caption and image border reflects the average UMAP color of all voxels in the cluster.
For examples voxel captions and images from various OFA and PPA voxels-alow with their corresponding quantitative metrics compared across three models (LaVCa, BrainSCUBA, Top-1), see Figures
A15 A16
and A17
4.3 ROI-LEVEL DIVERSITY ANALYSIS
Our results thus far demonstrate that LaVCa produces accurate voxel captions than BrainSCUBA and better captures both inter- and intra-voxel diversity. We next ask whether LaVCa can reveal richer representational content inside ROIs that earlier neuroimaging studies have largely described as selective for simpler categories- for example, faces in the OFA or places in the PPA (Gauthier et al., 2000; Haxby al. 2000 Epstein & Kanwisher] 1998). We conduct qualitative and quantitative evaluation using La VCa's captions and generated images to analyze diversity that exists beyond the known selectivity in the ROI.
Qualitative Assessment. We explore the semantic diversity LaVCa captions across four ROIs (EBA, OFA, PPA, OWFA) by applying UMAP to their Cexp-Text embeddings and visualizing the resulting distributions on a flatmap (Figure 5h. top). each ROI, we observe a broad spectrum of UMAP colors. indicating multiple meaningful clusters within regions known for distinct category- selective responses. The presence of this broad spectrum is consistent across participants (Figure A18
AT9
8

---

Published as a conference paper at ICLR 2026
Table 2: Average prediction accuracy with standard error across subjects when captions within each ROI are shuffled (Shuffled) versus used as is (Original).
Body areas
Face areas
Place areas
Word areas
Model
EBA
FBA-2
OFA
FFA-1
OPA
PPA
OWFA
VWFA-1
Shuffled
0.018=0.008
0.018=0.005
0.02820.004
3.016=0.003
0.11620.024
0.151=0.028
0.025=0.000
0.03420.009
Original
0.157=0.005
0.125=0.010
0.095:0.009
0.111=0.003
0.200:0:002
0.213:0.027
0.084=0.013
0.158:0.007
Across ROIs, we observe diverse nouns and verbs that not only align with prior selectivity profiles but also reveal richer, voxel-level selectivity for object categories and actions. Both EBA and OFA frequently include common nouns such as people" and person," and the verb distributions highlight ROI-specific action tendencies: EBA is enriched for body-related actions (e.g., hold"), whereas OFA is enriched for face-related actions (e.g., smilc"). These patterns are consistent across participants (Figure A18 A19).
Finally, to highlight how each caption and its corresponding voxel image relate to specific colors in semantic space, we project them onto flatmap (Figure (5p). We divide the samples into eight clusters by labeling each of the three UMAP dimensions as "High" (2 2/3) or "Low" (S 1 1/3.. From each cluster, pick the voxels with the highest prediction accuracy (or one if only one qualifies, or none if none qualify) and illustrate their captions and generated images.
In OFA, some captions related to faces (e.g., "face," "person," "animal"), while particular voxels encoded more fine-grained features such as "eye," " "tongue," or "smiling," " and other voxels encoded information like "animal," "bear," "cardinal." " Thus, even within this ROI, there appears to be substantial functional differentiation among inter-voxel that extends beyond generic "face" category.
Moreover, we observe nutra-voxel diversity, where single caption incorporates multiple ideas (c.g.. "A food packaging features a smiling person and a cartoon character"), "), suggesting that individual voxels can simultaneously encode several distinct concepts. These findings highlight the fine fine-grained functional specialization across inter-voxel within the ROI and the diverse nature of intra-voxel encoding beyond singular concepts.
The results for all participants, visualizing the top two captions for each cluster directly in the UMAP space, can be found in Figures A20 A21 A22
and A23
Quantitative Assessment. We next determine how many distinct captions appear in each ROI by comparing the sentence-level prediction accuracy of each ROI when captions are maintained in their original form versus shuffled within the ROI (Table [2). For each category (body, face, place, and word area), we select two ROIs with the largest total voxel count across all subjects, resulting in eight ROIs in total. all ROIs, shuffling reduces prediction accuracy significantly. For example, in the OFA, accuracy drops from 0.0945 (Original) to 0.0280 (Shuffled), 3.3-fold decrease; in the PPA, accuracy falls from 0.213 (Original) to 0.151 (Shuffled), a 1.4-fold decrease. Thus, even in regions traditionally linked particular concepts, voxels exhibit a range of distinct selectiittees..... Furthermore, the average caption similarity between the same ROIS of different subjects is relatively high at 0.227, compared to 0.171 between different ROIS of different subjects, indicating that such diversity is reproducible across subjects (Figure AIl).
Next, we quantify how many different semantic concepts a single voxel can encode (i.e., its degree of multi-concept selectivity). We perform the following analysis: (1) extract every unique noun from all voxel captions within the ROI; (2) obtain Cexp-Text embeddings for each noun using the prompt "A photo of w."rd)." and cluster them with k-means (k 6); (3) for each voxel, count how many of its nouns fall into different clusters. Across all ROIs, we find that most voxels are associated with multiple clusters, indicating multi-concept selectivity (Table A12). Thus, even within ROIS whose vocabulary is relatively coherent, individual voxels can encode several distinct concepts. Furthermore, by aggregating the nouns used in this clustering analysis from all subjects and examining the extent to which each subject's 's voxels belong to the subject-shared clusters, we evaluate the cross-subject reproducibility of ROI diversity (Figure A12). In both the OFA and PPA, voxels from all subjects populate the same clusters, suggesting that such diversity is, to some extent, consistent across individuals.

---

Published as a conference paper at ICLR 2026
5 DISCUSSION & CONCLUSIONS
In this study, we introduce a novel method called LaVCa, which leverages LLMs to produce data- driven, natural-language descriptions of voxel selectivity in the human visual cortex. The voxel captions generated by LaVCa exhibit higher accuracy and greater semantic diversity than those generated by the existing approach, BrainSCUBA. We attribute this improvement to our mechanism for integrating multiple keywords extracted by advanced LLMs, which enables a more comprehensive capture of the diverse selectivity patterns across voxels. Furthermore, LaVCa uncovers richer representational content within ROIs that earlier neuroimaging studies had characterized as selective for simpler categories. By revealing that even "category-sslecte"
areas such as the OFA and PPA encode broader spectrum of concepts, our findings challenge long-standing assumptions about functional specialization in the visual cortex. See Sections A.6 and A.7 for the Limitation and Impact Statement.
ACKNOWLEDGMENT
Y.T. was supported by JST PRESTO (Grant Numbers JPMJPR2316 and JPMJCR2574) and ISPS KAKENHI (Grant Numbers JP19H05725 and JP24K02999). S.N. was supported by JSPS KAKENHI JP23H05493 and JP24H00619 and JST JPMJCR24U2.
ETHICS STATEMENT
This study did not involve the collection of any new neural recording data. Instead, we relied exclu- sively the Natural Scenes Dataset (NSD), which is openly accessible to the research community. The dataset can obtained from ht httptt:////aatttte sce.esssasseeeeeeeesttste... org/ subject to their terms of use.
We conducted all analyses on this publicly released dataset and did not handle any personally identifiable information. Based on the nature of the data and the scope of our methods, we do not anticipate harmful applications this work.
REPRODUCIBILITY STATEMENT
We have made substantial efforts ensure the reproducibility of our results. Details of the LaVCa pipeline, including model architecture, training procedure, and evaluation metrics, are described in Section 3 Additional implementation details, hyperparameter settings, and preprocessing steps for the Natural Scenes Dataset (NSD) are provided in the Appendix. Furthermore, the source code and scripts necessary to reproduce our experiments are publicly available at: https: tig.............g com/ suyamat/LaVC. L/L.Ca
LLM USAGE
In accordance with the ICLR policy on the use of large language models (LLMs), we report that LLMs were employed exclusively for language-related assistance. Specifically, used LLMSs to aid the translation of text into English and to polish the grammar and style of the manuscript. All research ideas, experimental design, data analysis, and scientific interpretations were conceived and conducted entirely by the authors.
The use of LLMs did not contribute the formulation of research questions, methodology, or conclusions. The authors take full responsibility for the final content of the paper, including all text that was assisted by LLs-based tools.
REFERENCES
Emily J Allen, Ghislain St-Y St-y,es, Yihan Wu, Jesse L Breedlove, Jacob S Prince, Logan T Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, et al. A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence. Nature neuroscience, 25(1): ): 126126, 2022.
10

---

Published as a conference paper at ICLR 2026
Richard Antonello, Aditya Vaidya, and Alexander Huth. Scaling laws for language encoding models in fmri. Advances in Neural Information Processing Systems, 36, 2024.
Nicholas Bai, Rahul A Iyer, Tuomas Oikarinen, and Tsui- Wei Weng. Describe-and antebe--nnddisseee In- terpreting neurons in vision networks with language models. arXiv preprint arXiv:2403.13771, 2024.
David Bau, Bolei Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 6541-6544 2017.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020. 2020.
Kirill Bykov, Laura Kopf, Shinichi Nakajima, Marius Kloft, and Marina Höhne. representations with inverse recognition. Advances in Neural Information Processing Systems, 36, 2024.
Labeling neural
Nadine Chang, John A Pyles, Austin Marcus, Abhinav Gupta, Michael Tarr, and Elissa M Aminoff. Bold5000, public fmri dataset while viewing 5000 visual images. Scientific data, 6(1):49, 2019.
Jiaxuan Chen, Yu Qi. Yueming Wang, and Gang Pan. Bridging the gap between brain and machine in interpreting visual semantics: Towards self-adaptive brain-to-text decoding, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 11938-21948, 2025a.
Jiaxuan Chen, Yu Qi, Yueming Wang, and Gang Pan. Mindgpt: Interpreting what you see with non-invasive brain recordings. IEEE Transactions on Image Processing, 2025b.
Lisa Dunlap, Yuhui Zhang, Xiaohan Wang, Ruiqi Zhong, Trevor Darrell, Jacob Steinhardt, Joseph E Gonzalez, and Serena Yeung-Levy. Describing differences in image sets with natural language. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24199-24208, 2024.
Russell Epstein and Nancy Kanwisher. A cortical representation of the local visual environment. Nature, 392(6676):598 601, 1998.
Isabel Gauthier, Michael J Tarr, Jill Moylan, Pawel Skudlarski, John C Gore, and Adam W Anderson. The fusiform "face area" part of network that processes faces at the individual level. Journal of cognitive neuroscience, 12(3):495-504, 2000.
Umut Güçlü and Marcel AJ Van Gerven. Deep neural networks reveal a gradient in the complexity of neural representations across the ventral stream. Journal of Neuroscience, 35(27); 10005-10014, 2015.
James V Haxby, Elizabeth A Hoffman, and M Ida Gobbini. The distributed human neural system for face perception. Trends in cognitive sciences, 4(6):223-233, 2000.
Evan Hernandez, Sarah Schwettmann, David Bau, Teona Bagashvili, Antonio Torralba, and Jacob Andreas. Natural language descriptions of deep visual features. In International Conference on Learning Representations, 2021.
Nhat Hoang-Xuan, Minh Vu, and My T Thai. Lim-assisted concept discovery: Automatically identifying and explaining neuron functions. arXiv preprint arXiv:2406.08572, 2024.
Alexander G Huth, Shinji Nishimoto, An T Vu, and Jack L Gallant. A continuous semantic space describes the representation of thousands of object and action categories across the human brain. Neuron, 76(6):1210 1224, 2012.
Alexander G Huth, Wendy A De Heer, Thomas L Griffiths, Frédéric E Theunissen, and Jack L Gallant. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature, 532 (7600):453 458, 2016.
11

---

Published as a conference paper at ICLR 2026
Neha Kalibhat, Shweta Bhardwaj. C Bayan Bruss, Hamed Firooz, Maziar Sanjabi, and Soheil Feizi. Identifying interpretable subspaces image representations. In International Conference on PMLR, 2023.
Machine Learning, pp.
Kendrick N Kay, Thomas Naselaris, Ryan Prenger, and Jack Gallant. Identifying natural images from human brain activity. Nature, 452(7185):352-333) 2008.
Alina Kuznetsova, Hassan Rom, Neil Alldrin, Jasper Uijlings, Ivan Krasin, Jordi Pont-Tusel Shahab Kamali, Stefan Popov, Matteo Malloci, Alexander Kolesnikov, Tom Duerig, and Vittorio Ferrari. The open images dataset v4: Unified image classification, object detection, and visual relationship detection at scale. IJC,, 2020.
Tom Dupré La Tour, Michael Eickenberg, Anwar O Nunez-Elizalde, and Jack Gallant. Feature- space selection with banded ridge regression. Neurolmage, 264: 2022.
Amanda LeBel, Lauren Wagner, Shailee Jain, Aneesh Adhikari-Desal Bhavin Gupta, Allyson Morgenthal, Jerry Tang, Lixiang Xu, and Alexander G Huth. A natural language fmri dataset for voxelwise encoding models. Scientific Data, 10(1):555, 2023.
Mark Lescroart and Jack L Gallant. Human scene-selective areas represent 3d configurations of surfaces. Neuron, (1):11777 19211997711111 2019.
Zhuang Li, Yuyang Chai, Terry Yue Zhuo, Lizhen Qu, Gholamreza Haffari, Fei Li, Donghong Ji, and Quan Hung Tran. Factual: benchmark for faithful and consistent textual scene graph parsing. arXiv preprint arXiv:2305.1749) 2023.
Andrew F Luo, Margaret M Henderson, Michael J Tarr, and Leila Wehbe. Brainscuba: Fine fine-grained natural language captions of visual cortex selectivity. arXiv preprint (xXiv:3310004422) 2023.
Andrew F Luo, Jacob Yeung, Rushikesh Zawar, Shaurya Dewan, Margaret M Henderson, Leila Wehbe, and Michael Tarr. Brain mapping with dense features: Grounding cortical semantic selectivity in natural images with vision transformers. arXiv preprint arXiv:2410.05266 2024.
Ron Mokady, Amir Hertz, and Amit H Bermano. Clipcap: Clip prefix for image captioning. arXiv preprint arXiv:2111.0911 1190944, 2021.
Jesse Mu and Jacob Andreas. Compositional explanations of neurons. Advances in Neural Information Processing Systems, 33:17153-17161 2020.
Yuko Nakagi, Takuya Matsuyama, Naoko Koide-Majima, Hiroto Q Yamaguchi, Rieko Kubo, Shinji Nishimoto, and Yu Takagi. Unveiling multi-level and multi-modal semantic representations in the human brain using large language models. bioRxiv, pp. 2024-02, 2024.
Tomoya Nakai and Shinji Nishimoto. Quantitative models reveal the organization of diverse cognitive functions in the brain. Nature communications, 11(1):1142, 2020.
Tomoya Nakai and Shinji Nishimoto. Representations and decodability of diverse cognitive functions are preserved across the human cortex, cerebellum, and subcortex. Communications Biology, 5(1): 1245, 2022.
Thomas Naselaris, Kendrick N Kay, Shinji Nishimoto, and Jack L. Gallant. Encoding and decoding in fmri. Neuroimage, 56(2):400 2011.
Shinji Nishimoto, An T Vu, Thomas Naselaris, Yuval Benjamini, Bin Yu, and Jack L Gallant. Reconstructing visual experiences from brain activity evoked by natural movies. Current biology. 21(19):16411164, 2011.
Tuomas Oikarinen and Tsui-Wei Weng. Clip-dissect: Automatic description of neuron representations in deep vision networks. arXiv preprint arXiv:2204. 10965, 2022.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp. PMLR, 2021.
12

---

Published as a conference paper at ICLR 2026
Nils Reimers and Iryna Gurevych. bentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1906 10084, 2019.
Martin Schrimpf, Idan Asher Blank, Greta Tuckute, Carina Kauf, Eghbal A Hosseini, Nancy Kan- wisher, Joshua Tenenbaum, and Evelina Fedorenko. The neural architecture of language: Integrative modeling converges on predictive processing. Proceedings of the National Academy of Sciences, :18445):21105661111 2021.
Christoph Schuhmann, Romain Beaumont, Richard Vencu, Cade Gordon, Ross Wightman, Mehdi Cherti, Theo Coombes, Aarush Katta, Clayton Mullis, Mitchell Wortsman, et al. Laion-5b: An open large-scale dataset for training next generation image-text models. Advances in Neural Information Processing Systems, 0012222777222999 2022.
Chandan Singh, Aliyah R. Hsu, Richard Antonello, Shailee Jain, Alexander G. Huth, Bin Yu, and Jianfeng Gao. Explaining black box text modules in natural language with language models, 2023. URL https: /httts:///attiit:
org/ hbtp///s.//30// 09863
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie- Van Liu. Mpnet: Masked and permuted pre-training for language understanding. Advances in neural information processing systems, 33: 16857-16867, 2020.
Yu Takagi and Shinji Nishimoto. High-resolution image reconstruction with latent diffusion models from human brain activity. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14453-14463, 2023.
Jerry Tang, Amanda LeBel, Shailee Jain, and Alexander G Huth. Semantic reconstruction of continuous language from non-invasive brain recordings. Nature Neuroscience, 26(5):858-866, 2023.
Michael Tschannen, Alexey Gritsenko, Xiao Wang, Muhammad Ferjad Naeem, Ibrahim Alabdul- mohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Xia, Basil Mustafa, et al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arxiv preprint 2rXiv:2502.147866 2025.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc Le, Denny Zhou, al. Chain-of-thought prompting elicits reasoning in large language models. Advances in neural information processing systems, 35:24824-24837, 2022.
Tung-Yo Yu Wu, Yu- Yu-Xiang Lin, and Tei Weng. And: Audio network dissection for interpreting deep acoustic models. arXiv preprint arXiv:2406. /6990, 2024.
Chunyu Xie, Bin Wang. Fanjing Kong, Jincheng Li, Dawei Liang, Gengshen Zhang, Dawei Leng, and Yuhui Yin. Fg-elip: Fine-grained visual and textual alignment. arXiv preprint 205502250,.0577, 2025.
Yuan Yao, Tianyu Yu, Ao Zhang, Chongyi Wang, Junbo Cui, Hongji Zhu, Tianchi Cai, Haoyu Li, Weilin Zhao, Zhihui He, Qianyu Chen, Huarong Zhou, Zhensheng Zou, Haoye Zhang, Shengding Hu, Zhi Zheng, Jie Zhou, Jie Cai, Xu Han, Guoyang Zeng, Dahai Li, Zhiyuan Liu, and Maosong Sun. Minicpm-v: A gpt-4v level milm on your phone. arXiv preprint 2408,01800 01800, 2024.
Zequn Zeng, Yan Xie, Hao Zhang, Chiyu Chen, Bo Chen, and Zhengjue Wang. Meacap: Memory- augmented zero-shot image captioning. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pp. 14100-14110, 2024.
13

---

Published as a conference paper at ICLR 2026
A APPENDIX
A.1 FULL RELATED WORK
A.1.1 INTERPRETING THE REPRESENTATIONS OF THE BRAIN'S NEURONS.
Encoding models have long been used in neuroscience to interpret neural representations within the brain (Kay et al. 2008 Nishimoto et al. 2011 Naselaris et al.) 2011 Huth al. 2012). These studies used interpretable features, such as low-level visual attributes. or high-level semantic features, such one-hot encoding of words, for straightforward voxel-wise interpretation.
Recent approaches use features derived from DNNs and have demonstrated higher explanatory power for brain activity than those using simpler. more interpretable features (Güçlü Van Gerven 2015) Schrimpf et al. 2 (Takagi & Nishimoto [023 Antonello et al., (024). However, the interpretability of these DNN-based encoding models remains challenging, leading to the development of methods that condense the entire set of voxels into small number of universal and interpretable axes (Huth)
ct al. 2016 Lescroart & Gallant 2019 Nakagi et al. 2024).
Recent approaches propose data-driven methods to describe the properties of individual brain voxels using natural language (Luo et al., 2023 Singh et al., 2023) when analyzing brain representations at finer, voxel-wise level. BraiSCUBA (Luo al. (022) is end-to-end method that uses an existing image captioning model, which provides rosel-wise captions of the visual cortex in a data-driven manner. BrainSCUBA projects each voxel's 's encoding weight onto the image feature space via dot-product attention, identifies regions of highest similarity, and then uses a text decoder to generate captions describing the images to which the voxel is most selective. This approach provides a data-driven natural natural-language -laguagee description of voxel selectivity without additional training. Similarly, SASC (Singh et al. 2023) uses IMRI data collected during speech listening (LeBel al., 2023 to identify the short phrases that most strongly activate each voxel. It then uses an LLM to combine these short phrases into a single, data-driven caption describing each voxel's s semantic properties.
Our proposed method also generates data-driven voxel captions but differs in several ways. First, BrainSCUBA constrained to pre-existing, end-to-end image captioning models. In contrast, our approach divides the task into (i) identifying an optimal set of images and (ii) converting these images into a caption, allowing us to use any vision model aligned with language and any LLM with advanced language capabilities without requiring specialized fine-tuning. Furthermore, although SASC uses an LLM to create voxel captions, primarily synthesizes short, cow-information phrases (e.g., trigrams), producing only simple keyword-based captions. In contrast, method summarizes more diverse and informative text and then uses these extracted keywords to compose a complete sentence, capturing richer range of voxel-level properties.
A.1.2 INTERPRETING THE REPRESENTATIONS OF ARTIFICIAL NEURONS DNNs
Interpreting artificial neurons is key challenge in understanding how DNNs process information. We can potentially examine human neural representations at a finer granularity by applying the data-driven and highly accurate interpretation methods developed for artificial neurons analyze human brain voxels.
Numerous studies have aimed associate artificial neurons with human-interpretable concepts (Bau et al. 2017; Mu & Andreas 2020; Oikarinen & Weng. 2022] Kalibhat et al.) 2023 Bykov et al. 2024). These methods link neurons to textual concepts by comparing neuron output feature maps with outputs from segmentation models. However, these approaches are constrained by predefined concept sets or limited to the dataset's words and phrases. MILANN (Hernandez et al., (202T) introduced generative approach, enabling adaptation different domains and tasks, but requires annotated data, which poses challenges for scalable applications.
LLMs permit open-ended descriptions of artificial neurons without additional model training (Singh et al. 2023, Bai et al.) 2024 Wu et al. 2024 Hoang-Xuss Xuan et al. 2024). Analogous to these methods, study also leverages LLMs to generate open-ended concepts for brain neurons rather than artificial neurons, seeking flexible and diverse interpretations that do not depend on predefined vocabularies.
14

---

Published as a conference paper at ICLR 2026
A.2
IMPLEMENTATION DETAILS
A.2.1
GENERATING VOXEL CAPTIONS
In this study, leverage the "Sentence Composer" proposed in the image captioning model Mea- Cap (Zeng et al. 2024) referred to as the seyyords-to-sentent LM" in the original paper-to generate centence-level captions from keywords.
Notation
K fli,.... km J: keyword set extracted by an LLM.
W Rd: voxel-wise encoding weight.
(a,...., J: top-k captions of the voxel's optimal images.
dr(:): CLxP-Text embedding operator.
u
sim(u, v)
cosine similarity.
[
Iterative Decoding Procedure
We begin with an initial draft caption
(0)
[kc 12 ... km),
obtained by concatenating the keywords K in their given order (separated by spaces). Starting from this seed, CBART iteratively refines the caption through the following steps:
1. Action prediction. For each position in the current draft c(1), the encoder assigns of the actions COPY, REPLACE, or INSERT.
2. Candidate generation. At positions marked for replacement or insertion, the decoder proposes the top-n lexical candidates W; (w..t..., Wj,n) ranked by the token likelihood Po(w
Fluency (log Pg) ensures linguistic naturalness.
Image relevance grounds the sentence in visual evidence from the voxel's optimal images.
Voxel relevance (sim(or(w), W)) links the caption to the voxel's representation.
Each candidate Wj is scored by
1 k + d2
S(w) = >1 log Po(w c22
sim ( + A (m((r(w), (AI)
where set A2, As) (0.2, 0.2, 1.2).
3. Token selection and refinement. The token with the highest S(w) replaces or is inserted position j. yielding the updated draft (I++)) The loop repeats until every position is predicted as COPY, producing the final caption é.
Rationale
Fluency (log Po) encourages linguistic naturalness.
Image relevance 1 Σ (mm(ór(w) or()))) grounds the sentence in visual evidence &
drawn from the voxel 's optimal images.
Voxel relevance (sim ( (dT(w), W)) ties the caption the voxel's 's learned representation.
By jointly optimizing the score in equation Al the method transforms discrete keyword sets into coherent sentence that is linguistically natural, visually grounded, and specifically aligned with the voxel's 's weights W.
15

---

Published as a conference paper at ICLR 2026
Table A1: Pretrained checkpoints used in experiments.
Category
Model
Repository (Hugging Face)
CLIP
Contrastive VLM
SigLIP2
google/sigglip22ban http://gg...ppi.aae.aaaacaaaaaae
FG-CLIP
sthoo:50//3gcllip-bbuss
Dpetano/AMin.CCPP-Laaaa333222 MinCCPM-Llama--225
MLLM
BLIP
gpt-4o
N/A (OOOnAATAAP])
LLM
Llama 3.1-70B
Sentence-BERT MPNet
Sentence Similarity Model
Text-to-Image Model
FLUX. 1-schnell
A.2.2 STATISTICAL TESTING
For each voxel the observed prediction accuracy is quantified Spearman's rank correlation p To realize the null hypothesis of no association, the activity vector is randomly permuted B times, yielding surrogate correlations {Pem 16=1- Pooling across all N voxels produce a global null distribution. The one-tailed p-value is computed
# (mmill
pob p00 +1 }1
P. = BxN+1
We control for multiple comparisons using the Benjamini-Hochberg false-discovery-rate (FDR) procedure (a 0.05); voxels with g 0.05 are declared significant. our experiments, we set B = 1000.
A.2.3 PRETRAINED CHECKPOINTS
We rely variety of pretrained models for different components of our pipeline. Most of the models are publicly available checkpoints hosted on Hugging Face, including contrastive vision language models (CLIP, SigLIP2, FG-CLIP), multimodal LLMs
V2.5, BLIP). LLM (Llama 3. 3-1.,.B), sentence similarity models (Sentence-Bre) BBERT, MPNet), and a text-to-image model (FLUX.11 1- schnell). For keyword extraction, we gpt-4o, which not available checkpoint but is accessed via the OpenAl API. A complete list of all models and their repositories is summarized in Table Al
A.2.4 BRAINSCUBA
At the outset of our project in January 2025, the original BrainSCUBA codebase had not yet been released, so we implemented the method ourselves for this study. In BrainSCUBA, the encoding weights (linear layers) are learned using gradient descent. In our implementation, consistent with our proposed method, we trained the encoding weights using L2-regularized linear regression from himalaya library package La Tour al. 2022).
Moreover, BrainSCUBA projects each voxel's encoding weight into image space using a dataset of 2 million images, combining Openlmages (Kuznetsova et al. 2020) and LAION-A v2 (6+ sub- set) (Schuhmann et af., 2022). However, the specific images selected from each dataset are not disclosed. We ensure fair and dataset-indepeendet comparison by relying solely the 1.7 million images from OpenImages (the same dataset used by our proposed method, LaVCa). We leverage the training set of the subset that is accompanied by bounding boxes, object segmentations, visual relationships, and localized narratives.
For other hy hyperparameters, we tested temperature values of 1.0. 1/10. 1/100, 1/150 (the value used in the BrainSCUBA paper), and 1/500 for the softmax projection (Figure A1). We used beam search with beam width of 5 to generate the text decoder's caption as described in the BrainSCUBA paper.
hutp://gthhhh.cmmmlllaaalbma/mm
16

---

Published as a conference paper at ICLR 2026
subj01
subj02
0.30
0.15
0.20
Prediction accuracy (p)
0.10
10
10
169
0.15
10
10-1
109
Temperature
Temperature
subj05
subj07
0.25
LavCa
0.30
0.25
0.15
0.20
10
10
10"
0.10
10
10-1
10a
Temperature
Temperature
Figure A1: The change in accuracy of BrainSCUBA with respect to temperature. The error bars rep- resent standard error. Moreover, the star markers on the plot indicate the point where the temperature is set to 1/150, adopted in the original BrainSCUBA paper. The green line represents the average value of LaVCa.
A.2.5 USED COMPUTE RESOURCES
All experiments are conducted on single Lambda Labs cloud instance equipped with eight NVIDIA A100-SXM4 GPUs (40 GB each). The host system features dual-socket AMD EPYC 7542 processor, providing 124 logical CPU cores and 512 GB of DDR4 RAM.
In this setting. voxel-wise Ridge-regression training for subject finishes in -7 and occupies 5.4 GB of GPU memory. Loading the 1.7 M candidate images for optimal-image search, performed on the CPU, takes ~1,490 per subject. The subsequent per-voxel anpeline-optimalle multimodalLLM captioning, keyword extraction, and SentenceCompen per voxel with peak GPU footprint of 5.0 GB. Processing 20,000 voxels for single subject therefore requires -83 h end-tor end-to-end, which remains practical for offline analyses in systems neuroscience.
search, generation-runs in -15
17

---

Published as a conference paper at ICLR 2026
Table A2: Hyper-parameter settings for gradient-descent training.
Parameter
Value
Optimizer
AdamW (81 0.9, 82 0.999)
Batch size
64
Initial learning rate (7))
3 x 10-4
Final learning rate (71)))
1.5 x 10
Scheduler
Exponential decay
Decay factor (r))
(nr/no)) 1//50 0.87
Weight decay
2 x 10-2
Maximum epochs
50
Stopp-gg patience
5 epochs
Table A3: Comparison of prediction accuracy for the encoding model. For each subject, the mean + standard error the test set is reported for the top 5,000 voxels that achieve the highest accuracy the train data.
Methods
ridge
gradient
layers
voxel-wise
weight-
subj01
subj02
subj0S
subj07
V
0.501
0.524=0.001
0,570+0.00
0.421=0.001
Linear
0.500=0 002
0,523=0.001
0.567=0.00)
0,420=0.00)
v
0.48420.002
0.512=0.002
0.56310.001
0.405e0.002
Nonlinear
d
0.492+0.002
0.516:0.001
0.56540.001
0.411=0.002
0.491
0.518=0.001
0.563+0.001
0.411=0.002
A.3
ABLATION STUDY
A.3.1
ENCODING MODEL
We conduct an ablation study to evaluate how hyper-parameter choices affect prediction accuracy for the encoding model (Table A3). Here, "accuracy" denotes the correlation coefficient obtained when predicting brain activity with the encoding model, rather than a voxel-caption based based metric).
Voxel-wise versus shared weights. We compare two variants of ridge regression: a voxel-wise version that learns individual weight vector for each voxel and shared-weights version that learns a single weight vector common to all voxels. The regularisation coefficient is sampled at 25 logarithmically spaced points from 10 to 20 and the optimal value is selected via five-fold cross-validation. This comparison reveals almost no difference in accuracy between the two variants.
Ridge regression versus gradient descent. We next contrast linear layers trained with ridge regression against those trained with gradient descent. The hyper-parameter settings for gradient- descent training are summarised in Table A2 During optimisation, the final 10 % of the training data are held out for validation, and early stopping is triggered if validation accuracy fails improve for five consecutive epochs. To ensure a fair comparison, ridge regression is fitted using only the first 90 % of the training data, so that both methods see an identical amount of data. Under the shared-weight setting, ridge regression outperforms gradient descent, indicating that an analytically derived linear solution is more effective for training the encoding model.
Linear versus non-linear models. Finally, we compare linear and non-linean architectures. For the non-linear networks, the hidden-layer width is set equal to the input dimensionality (512, cor- responding to the CLIP-Vision feature size). Introducing non-linearities yields higher accuracy than the single-layer linear model trained with gradient descent; however, increasing the depth from two to three layers offers no further benefit. Moreover, none the non-linear networks surpass the performance of ridge regression. These findings indicate that, although con-linear models trained with gradient descent can exceed their linear counterparts, they still fall short of the accuracy achieved by ridge regression.
18

---

Published as a conference paper at ICLR 2026
Table A4: Sentence-level prediction accuracy for 3-based versus encoding- moaggg-oodldaaaea based models. Values denote mean + standard deviation across test images for the top 5,000 voxels per subject, selected based on their training-ser prediction accuracy.
Condition
subj01
subj02
subj05
subj07
w/o Encoding model
0.201 + 0.082
0.248 + 0.086
0.276 + 0.096
0.094 + 0.054
w/ Encoding model
0.246 + 0.066
0.287 0.075
0.306 + 0.084
0.218 0.073
A.3.2 CAPTION GENERATION
We investigate how various hyper-parameter settings influence the accuracy of the voxel captions (Tables [44] A5). Unless otherwise specified, adopt the primary-analysis configuration: CLIP- Vision as the default contrastive VLM for feature extraction, 50 optimal images per voxel, MiniCPM-V as the MILM for captioning the optimal images, five extracted keywords, gpt-4o as the keyword extraction model, Semplaarbaseee prompting as the prompting strategy, and use of the Sentence Composer.
Effect the Encoding Model Our goal LaVCa is to characterise voxel selectivity at the level of DNN-derived visual features, rather than to interpret voxels directly from raw 8-values. Nevertheless, to verify that the encoding model indeed improves interpretabiltyy-by suppressing noise and non-visual components in the 3-responses and enabling better generalisation to novel stimuli- we performed an additional control analysis using a purely 3-driven baseline.
To directly assess whether the encoding model is necessary for identifying optimal images, conducted a control analysis in which, for each voxel, we selected the NSD images that elicited the strongest raw 3-values and generated voxel captions from their MS-COCO captions. Importantly, in this baseline the encoding-mode weights used in the later stages-key filtering and the Sentence Composer-are are replaced with uniform weighting scheme by setting all weights 1. This ensures that the entire pipeline operates without any feature-level information from the encoding model, providing an interpretation grounded solely in the measured 3-values.
This 3-based approach produced substantially lower predictive accuracy compared to LaVCa (Ta- ble A4.. Because this baseline relies solely on raw 3-values, is highly sensitive to measurement noise and non-visual components of brain activity, resulting in captions that generalised poorly to held-out images.
Contrastive VLM comparison. We next compare different contrastive vision-language mod- els (VLMs) used to extract latent features for voxel-wise encoding. Specifically, we evaluate SigLIP2 (Tschannen et al. 2025), FG-CLIP (Xie et al., 2025), and both the text and vision branches of CLIP (Radford et al. 2021). Ip. For CLIP-Text
obtain projection-layer embeddings from the COCO captions that were ass-inne to the NSD image stimuli. Overall, CLIP-based representa- tions (CL.I-Teext and CIIPo-Vsioo) achieve the highest accuracies. FG-CLIP performs comparably to CL.P-Vision, while SigL.IP does not surpass CLIP in our setting despite reports in the original SigLIP work that it often outperforms CLIP on benchmark tasks-yet its inclusion demonstrates that LaVCa generalises well across diverse VLM backbones.
Number of optimal images. We vary the number of optimal images used for keyword extraction from to 10, 50, and 100. Increasing the number up to 50 improves accuracy, presumably because relying only on top-ranked images can omit useful second- and third-ranked keywords. Using more images therefore captures a broader range of selective concepts. However, once the number of optimal images reaches 100, the improvement plateaus, likely because additional concepts can longer be adequately represented with only five keywords. These observations suggest that increasing the number of keywords, rather than merely adding more images, may further enhance accuracy.
Multimodal LLM comparison. We compare two multimodal LLMs for captioning the optimal images: the state-of-the- MiniCPM-V and the lighter, less accurate BLIP. MiniCPM-V slightly outperforms BLIP, indicating that more capable MILM can further improve LaVCa's voxel-caption
19

---

Published as a conference paper at ICLR 2026
Table A5:
Comparison of sentence-level brain activity prediction accuracy using different hyperpa- rameters. Accuracy is reported as the mean + standard deviation for the top 5,000 voxels in the test data, selected by training-ser accuracy.
Parameter
Setting
subj01
subj02
subj05
subj07
SigLIP2
0.232-10.065
0.27510.075
0.294 080
0.206110.070
FG-CLIP
0.244.10.070
0.285:10.076
0.306110.08
0.214100074
Contrastive VLM
CeiP-Text
0.246=+0.067
0.281=0.074
0.309 0.44
0.216=0.071
CLIP- Vision
0.246:+0.066
0.287=0.005
0.306 084
0.218=0.073
0.239:100068
0.279:L0.073
0.294+t0.08
0.209=0.073
0.268
0.281+00072
0.297++0.06 0.08
0.212--0.074
# optimal images
50
0,246++0.06
0.287+00.075
0.306+000084
0.218--0.073
100
0.2461100061
0.285
0.301 +0.083
0.215
MiniCPM-V
0,246 0.666
0.287
0.306 00.04
0.218
Multimodal LLM
BLIP
0.242 00688
0.285
0.302 0..04
0.215
0.237
0.274+0.073
0.295 085
0,207=0.072
# keywords
5
0,246 06066
0.287+00075
0.306 0.00
0.218=0.073
10
0.241
0.2791000074
0.296110.084
0.212-10.074
TextGraphParser
0.242+00.067
0.276+L0.033
0. 0.296 0.00
0.205
Extraction model
LJama3.
0.238-00.067
0.281+0.075
0.298++0.08
0.214=00073
gpt-4o
0.246110.066
0.287+0.005
0.306 084
0.218=0.073
Hidden CoT
0.239110.066
0.283
0.29810.00 083
0.21410.072
Prompt
Exemplar-based
0.246+00.066
0.287++0.075
0.306+0.084
0.218+0.073
0,246++0.066
0.287+00.075
0.306+ 0.004
0.218+00073
Sentence Composer
0.230++0.066
0.279+
0.296++00087
0.201
accuracy. Conversely, the modest gap between BLIP and MiniCPM-V suggests that our approach generalises well even with simpler captioning models.
Number of extracted keywords. With the number of optimal images fixed at 50, we vary the number of extracted keywords among 5, and 10. Increasing the output concepts from one five boosts accuracy, whereas extending the list to ten decreases accuracy-likely because irrelevant or noisy concepts introduced. The improvement five keywords indicates that voxels encode multiple concepts. but extracting too many can introduce noise. Thus, expanding the image set rather than the keyword count may be more effective strategy for capturing additional informative concepts.
Keyword-extraction model comparison. We evaluate three models for extracting keywords from the optimal image set: gpt-4o, 8-bit-quantised Llama 3.1-roB-Instruct, and the TextGraphParser Li al. (2023) employed in MeaCap. gpt-4o surpasses TextGraphParsee showing that an open-ended effective than simply pulling words from captions. It also exceeds Llama 3.1-70B-8bit, demonstrating that stronger LLMs can further raise accuracy. These results imply that LaVCa's interpretability will improve in tandem with future advances in LLM
L.M makes concept extraction
capability.
Prompt comparison. We compare two prompting strategies for summarising the captions as- sociated with each voxel's optimal image set: Exemplar-bused Prompting and a more structured Hidden CoT Prompting formulation. Exemplar-based Prompting follows the classical in-context learning paradigm (Brown et al. 2020 Dunlap et al.) 2024). where multiple positive and negative examples constrain the output format and guide the model toward producing concept-like captions. In contrast, the structured prompt incorporates chain-of thoutht instruction (Wei et al. (002),, encouraging the model to silently derive recurrent patterns before producing the final set of concepts.
expert role assignment and internal ("hidden"")
Across subjects, both prompting strategies performed similarly, indicating that LaVCa is robust to prompt specification. Nonetheless, Exemplar- based Prompting consistently achieved slightly higher
20

---

Published as a conference paper at ICLR 2026
Table A6: Comparison of sentence similarity models for accuracy evaluation. Accuracy is reported as the mean + standard deviation for the top 5,000 voxels in the test data, selected by training-set accuracy.
Model
subj01
subj02
subj05
subj07
MPNet 0.245+00067
0,287++0.075
0.305++0.088 0.306:10.084
0.216++0.069
Sentence-BERT
0.246-10006
0.287
0.218=100073
accuracies for all subjects, suggesting that explicit examples remain effective mechanism for stabilising the output structure this caption-concept abstraction task.
Effect of the Sentence Composer. Finally, we assess the Sentence Composer by comparing results with and without it. Incorporating the Sentence Composer yields higher accuracy than relying on keywords alone, suggesting that contextual information beyond isolated concepts enables a more fine-grained interpretation of voxel properties.
A.3.3 SENTENCE SIMILARITY MODEL FOR EVALUATION
We also examine how the choice of sentence similarity model used for evaluation affects voxel- caption accuracy (Table A6). Specifically, we compare MPNet (Song et al.] (2020) and Sentence- BERT (Reimers & Gurevych] (2019), both widely used models for computing sentence embeddings. Overall, the two models yield nearly identical accuracies across all subjects, with Sentence-BEi BERT performing marginally better. This consistency suggests that LaVCa's evaluation results robust to the particular choice of sentence similarity model, and that the observed improvements are not dependent on model-specine idiosyncrasies.
A.4 DATA-SIZE SENSITIVITY
In this appendix, we examine how LaVCa's cuption-predictton accuracy depends on the amount and structure of the stimulus data used within the NSD dataset. We vary (i) the number of stimulus images supplied to La and (ii) the number of stimulus-image categories used as part of the method's categorization stage.
A.4.1 SAMPLE- SIZE SENSITIVITY
To evaluate how the size of the NSD dataset influences LaVCa's performance, vary the number of stimulus images provided to the method across 1, 10, 100, 500, 1000, 5000, and approximately 9000 (the full NSD stimulus set). Figure A2 summarizes how prediction accuracy changes across these sample sizes.
The number of stimulus-image categories is fixed 80, based the COCO object categories, except when fewer than 80 images are used, in which case the number of categories matches the sample size.
La'Ca's accuracy shows a clear dependence on the number of available stimulus images. With only a small number of images (such as 1 or 10), accuracy decreases substantially because the limited variety of stimuli restricts the method's s ability to infer stable relationships between fMRI responses and visual content. As more images are incorporated, accuracy improves rapidly and stabilizes once approximately 500 1000 stimulus images are available. Increasing the stimulus dataset beyond this range yields only modest gains, and the performance with the full -9000-imaae set is only slightly higher than that achieved with 5000 images. These results indicate that roughly 500-1000 stimulus images sufficient for La to produce reliable voxel-level captions, without requiring the full scale of the NSD stimulus set.
A.4.2 CATEGORY-SIZA SENSITIVITY
We next analyze how the number of stimulus stimulus-image categories provided to LaVCa affects caption- prediction accuracy. Figure A3 A3 provides summary of this analysis. The number of categories is
21

---

Published as a conference paper at ICLR 2026
subj01
subj02
0.30
0.35
0.25
0.30
0.25
0.20
0.20
0.15
0.15
0.10
0.10
Prediction accuracy (p)
0.05
0.00
10"
10
0.00
10a
10-
Sample size (log-scale)
Sample size (log-scale)
subj05
subj07
0.30
0.35
0.25
0.30
0.25
0.20
0.20
0.15
0.15
0.10
0.10
0.05
0.05
0.00
0.00
10
10
10
10a
10
109
10
10a
10a
10a
Sample size (log-acale)
Sample size (log-scale)
Figure A2: Effect of sample size test prediction accuracy. Mean prediction accuracy the test data for the top 5000 voxels selected using training-data performance, plotted across different sample sizes (x-axis in log scale).
varied across 2, 10, 40, and 80 only during the training stage, while the test evaluation the full set of stimulus images.
LaVCa's accuracy increases the categories become more fine-grained Extremely coarse category sets, such as those with or 10 categories, compress variability across the stimulus set and lead reduced accuracy. Using 40 categories captures substantially more structure within the stimulus data and already approaches the performance of the full 80-category condition. The 80-category condition performs best, particularly higher-level visual areas where finer distinctions between visual stimulus types are beneficial.
A.5 ADDITIONAL FMRT DATASET: BOLD5000
A.5.1 DATASET AND PREPROCESSING
To assess whether LaVCa generalises beyond NSD, additionally evaluate our method on the BOLD5000 dataset, which contains fMRI responses from four subjects viewing approximately 5,000 natural images (Chang et al. 2019). Following our NSD pipeline, we first compute single-trial response patterns using GLMsingle 8-estimates for each session and concatenate them along the time dimension to obtain 4D volume for each subject.
For analyses, we restrict all experiments to the functional masks provided in BOLD5000, consisting of EirlyVis (early visual cortex), LOC (lateral occipital complex), OPA (occipital place area), PPA (parahippocampal place area), and RSC (retrosplenial cortex). Across these regions, the total number voxels amounts to approximately 2,500 per subject. Within each subject, we vectorise the GL Msingle 3-estimates into trial-wise response matrices and standardise each voxel by z-scoring
22

---

Published as a conference paper at ICLR 2026
subj01
subj02
0.245
0.290
0.285
0.240
0.280
0.270
0.230
0.265
Prediction accuracy (p)
0.220
0250
0.215 100
10
102
0.24
10
10
10a
Category size (log-scale)
Category aize (log-scale)
subj05
subj07
0.295
0,290
0.215
0.280
0.200
0.275
0.270
0.265
0.195
0.255
0.19
0.250
0.18 10
Category size (log-scale)
Category size (log-scale)
Figure A3: Effect of category size test prediction accuracy. Mean prediction accuracy on the test data for the top 5000 voxels selected using training-data performance, shown for different category sizes.
across all trials. Following the same procedure as in NSD, we first split the trials into 90% for training and validation and 10% for testing.
We then apply LaVCa to BOLD5000 using exactly the same configuration as in the main NSD experiments, without any dataset-specift tuning.
Because LaVCa evaluates voxel captions by comparing them with the ground-truth captions of the held-out stimuli, textual captions for the test images required. In BOLD5000, the only subset of stimuli accompanied by human-written captions is the set COCO images; therefore, test-set accuracy is computed exclusively on COCO trials using their official COCO captions and the corresponding 3 values. Importantly, the encoding model is trained on all available BOLD5000 trials, and only the evaluation stage is restricted COCO trials.
A.5.2 RESULTS ON BOLD5000
We apply LaVCa to the BOLD5000 dataset and compute voxel-caption accuracy on the COCO- captioned trials in the test split. Table summarises the caption periiit-preiitionn accuracy for the top 100, top 500, and top 1000 voxels in each subject, selected based on their training-ser -et performance.
Across all voxels, LaVCa achieved significant caption-prediction accuracy in roughly 40% of voxels.
Although La'Sa's performance on BOLD5000 is lower than on NSD, this reduction closely parallels the decrease in encoding-mode model accuracy between the two datasets. Specifically, the average encoding- model performance drops from 0.504 in NSD to 0.302 in BOLD5000 an approximately 40% reduction. The proportional decline in both encoding and caption-prediction the diminished performance reflects inherent differences in dataset quality (c.g., shorter scan durations and lower SNR in BOLD POLD5000), rather than overfitting of La to NSD.
prediction accuracy suggests that
23

---

Published as a conference paper at ICLR 2026
Table A7: Comparison of sentence-level caption prediction accuracy on BOLD5000 (subj01-subj04) "Top-N voxels" refers to the voxels with top-V prediction performance in the training data. Values are mean standard deviation on the test data.
Top-N voxels
subj01
subj02
subj03
subj04
Top-100
0.2655 0.0609
0.1927 + 0.0656
0.1525 + 0.0540
0.2117 0.0576
Top-500
0.1652 + 0.0780
0.1138 1 0.0737
0.1185 + 0.0560
0.1523 1 0.0668
Top-1000
0.1286 = 0.0769
0.0776 = 0.0749
0.1006 = 0.0575
0.1221 = 0.0743
Taken together, these results demonstrate that LaVCa generalises to a dataset with different subjects and stimulus sets.
To assess the reproducibility of voxel-level semantic organization across datasets and subject groups, we conducted WordCloud-base analysis analogous to that used in the NSD experiments (Fig- ure A4).
Across ROIs, the BOLD5000 results exhibit semantic patterns that closely parallel those observed in NSD despite differences in subjects, scanner parameters, and stimulus sets. In BOLD5000 LOC, which is broadly associated with object- and body-related processing. we observed frequent person- related nouns such person, reproducing the person-related cluster identified in NSD's s EBA. Verb distributions show similar correspondence: BOLD5000 LOC contained action verbs such as wear and hold, mirroring the body- and interaction-related action clusters prominent in NSD's EBA and OFA.
Scene-selective regions demonstrated an even stronger cross-dataset alignment. BOLD5000 OPA, PPA, and RSC all exhibited WordClouds enriched with place-related nouns including room, kitchen. These terms match the dominant scene-related cluster observed in NSD PPA, indicating that the underlying semantic structure of these ROIS is stable across datasets.
Taken together, these results show that the semantic clusters identified by LaVCa are not specific to particular dataset. Instead, similar diversity patterns emerge across both N$D and BOLD5000: object- and body-related concepts appear in lateral occipito-temporal regions, and place-related concepts are consistently represented in medial scene-selective regions. This cross-dataset reproducibility directly addresses the reviewer's question regarding the stability of semantic clusters across large populations and distinct image sets.
A.6 LIMITAION
Despite the overall improvement in brain activity prediction, we observe that face-selective regions do not achieve accuracy levels high as those in other ROIs (Figure 2). One reason may be that our current approach uses a Multimodal LLM (MLLM) to produce relatively simple captions for optimal images, often omitting important local features (e.g., "eyes," "nose"") and focusing more global terms (e.g.. "face," " "person"). Consequently, the subsequent summarization step lacks access these local details. Because our method relies on language descriptions, it has inherent limitations in capturing the fine-grained, local selectivity of these voxels. Incorporating recent techniques that visually interpret local voxel selectivity (Luo et al. 2024) could help address this gap.
Furthermore, while our current study describes voxel selectivity primarily in response to visual stimuli in the occipital cortex, there exist "multimodal voxels" in the brain that are simultaneously activated by auditory and linguistic information. and higher-order cognitive processes such as calculations, memory retrieval, and reasoning (Nakai & Nishimoto 2020 2022). Designing stimuli and experimental tasks encompassing diverse sensory inputs (e.g., auditory, textual) and cognitive challenges (e.g., recalling past events, performing reasoning tasks) is essential when interpreting such voxels. Because our approach uses LLM-bbsed textual summarization, it can be adapted to represent a wide range of stimuli and cognitive states in text form, providing a unified framework for multimodal integration. Looking ahead, by jointly modeling images, semantic information, auditory features, and cognitive tasks, we anticipate capturing the brain't 's integrated representation of both sensory and higher-order cognitive functions with greater accuracy.
24

---

Published as a conference paper at ICLR 2026
LOC (BOLD5000)
subj01
subj02
subj03
subj04
WOOT De Enteractioo ON
verb noun
car
pe sonn
0
woman
dog
N/A
N/A
Turon
poople
anima
O
än anima
#item
weall
peat SECATN
weal
dor initaa use
pose
inter
dianee
N/A
N/A
holo
set pore
no
OPA (BOLD5000)
subj01
subj02
subj03
subj04
furni
ture
ooffice
Bustdin group PSA
ffrn tur Cae
kiechen
verb noun
peepr
Toom
dow wattrr ter
space scene
WIndOW
group
oomm
FOOm
street
1OOm DOmEE
UnterAO
LEe
Window
space
O f
table scene SNE office
look
Sitra
sierl
seat sttnnnd
Frail
Wor
Cuse
wilh
e
take
park
interact
Sear
wait
weal
WO
shil
dance
shop
sit
WOOK
bake din work
gather lunes Time
interact
sit wil
p wait
T to
walk
wa lik
wail ee
ndhinn
work
US
PPA (BOLD5000)
subj01
subj02
subj03
subj04
verb noun
Jntector
O
kirchen
e
Ewindow
furnit
oom
buitding pooro ty
Om
interior
effice
Foom
spa
Foom
space
S spacc
building
furniture
furniture e
space
FIG
Lfffice oppp
ShOP
decorateworke
seat
din
waitt
gather Wler
wa
Light
S
in act
Indoor
in
Tail
Liveseat din
S
se
kulkpark K
C
SALE
WOOD
OF
Moreere githe
bandon
shop
wor
din
engaggaa
kitchen
Set
sea
Wait
muse
park
Se
oold
walkS
pose
indoon
Live
wa
swan
wooden
to
seat
din
Kine
park
n
shop
t
RSC (BOLD5000)
subj01
subj02
subj03
subj04
people
roon
Tibrary
people
traing
people
verb noun
S
Lee
veei cl
de
space
N/A
ste Feet
but
ConstructLorbuuic
REPLER
Scene
S
Scene
vehicle
seeem ongage
wo
study
K
shop
set wark senters Jib
parkwait
shöp
wait
outdoor
set
F
ga k
butio
N/A
Intergo
Wakk
render
wait
set
Wa
sit
di
luse
indoor
wearcrond
Figure A4: Word clouds depicting the 100 most frequent nouns and verbs appearing in voxel captions for the BOLD5000 dataset. Each panel shows the word cloud for particular brain region and subject.
A.7 IMPACT STATEMENT
We introduce a data-driven method that uses large language model to generate natural language captions of voxel-level visual selectivity. Using the method detailed in this paper, aim provide
25

---

Published as a conference paper at ICLR 2026
more fine-grained understanding of human visual function than previously achieved. We acknowledge that this human brain research could raise concerns regarding individual privacy. Although the present study examined relatively coarse-grainet voxel-level data, cannot dismiss the possibility that future advances measurement and analysis techniques may enable the extraction of more detailed individual-specinn information. In any case, obtaining explicit informed consent from participants remains crucial when collecting and using human brain activity data, as with the NSD dataset used in this study.
26

---

Published as a conference paper at ICLR 2026
Prompt
The following are the result of captioning a group of images:
"Three people a dark room with masks and lights."
"A band is performing on stage with various instruments and lights."
"A rock band is performing
stage with red Coca- Coca-Cola logo."
"The image depicts group people singing together room."
"A band performing on stage with a singer holding a microphone and a guitarist playing his instrument."
am a machine learning researcher seeking to elucidate the concepts of this group order to better understand my data.
Come up with 5 distinct concepts that are likely to be true for this group. Please write list of captions separated by bullet points ('""), For example:
dog next harse"
"a car in the rain"
"low quality" cars from side view"
"people in intricate dress"
" joyful atmosphere"
not talk about the caption, e.g.. "caption with one word" and not more than one concept. Also use singular form unless the concept naturally involves multiple objects.
The hypothesis should be caption, hypotheses like "more of "presence ...". "images with not provide multiple options by
are incorrect. Also do not enumerate possibilities within parentheses. using 'or' or 7 to maintain clarity. Here are examples of bad outputs and their corrections:
INCORRECT:
"various nature environment like lake, forest, and mountain" CORRECTED: "nature"
INCORRECT: image caption household object (e.g. bowl, vacuum, lamp)" CORRECTED:
household object'
INCORRECT: "Presence of baby animal" CORRECTED: "a baby animal"
INCORRECT: "Different types of vehicles including cars, trucks, boats, RVs" CORRECTED:
vehicle"
INCORRECT: "Image caption involving interaction between humans and animals" CORRECTED:
"interaction between humans and animals"
INCORRECT: "More realistic image" CORRECTED: "realistic image"
INCORRECT: "Insect (cockroach, dragonfly, grasshopper]" CORRECTED: "a insect"
INCORRECT: "newspaper magazine" CORRECTED: "a print media"
Again, want to identify the characteristics of this group. List properties that hold more often for the images in this group. Answer only with list (separated by bullet points **"). Your response:
Figure A5: The prompt used for summarizing the captions of optimal image groups with an LLM (Exemplar-basee Prompting). The text in red represents the captions of the optimal image group, which depend on the target voxel and the number of optimal images used. The blue number specifies the required number of concepts, which was varied during the ablation study.
27

---

Published as a conference paper at ICLR 2026
Prompt
expert
concept discovery
The following are captions produced for many images from the same unknown group:
"Three people dark
lights."
"A band performing on stage with varlous instruments and lights."
"A rock band is performing a stage with a red Coca-Cor Cola logo." "
"The image depicts group of people singing together
"A band performing on stage with singer holding a microphone and guitarist playing his Instrument."
essisting machine learning order to better understand the data.
trying identify underlying concepts by group images
Your internal reasoning process (which should not output]: Silently scan ell captions and infer recuriig petterns such se:
common objects,
common actions or interactions,
shared
environments,
shared moods visual styles.
Propose candidate concepts could explain why images belong group.
From these candidates, select 6 concepts that:
likely
many images this group (not just single example).
are mutually distinct (avaid strong overlap or near synonyms),
preferably cover concrete content (e.g., objacts, activites,, scanes) more abstract properties (e.g.. mood, style).
Output requirements (what you actually output]:
Output only exectyy hypotheses, each
separeted by butlet points
Each hypothesis must be written as single short caption (about 3-10 words).
noncertteturaltt involves multiple Do not talk about the caption itself (e-g., avoid phroses Like "caption with "..". hypothesis should be caption-like phrase, so do not use meta-rorms such "more of.... "presence of ...', "Images
singular form
with....
Do enumerate options, use parenthases,
" Inside single hypothesis.
Here are examples of bad outputs and thair corractions:
INCORRECT:
landscape"
-NCOORECT:
'a image caption of household (eject bowl, vacuum, lamp)' CORRECTED: "a household object"
baby 'Different types of vehicles including cars, trucks, boats, and RVs" CORRECTED; "a vehicle"
CORRECTED: baby animal"
INCORRECT:
INCORRECT: "Image caption involving interaction between humans and animals" CORRECTED: "interaction between humans and
animels'
INCORRECT: "More realistic image" CORRECTED: "realistic image"
INCORRECT: "insect (cockrosch, dragonfly, grasshoppern" CORRECTED: insect"
INCORRECT: "newspaper or magazine' CORRECTED: "a print media'
Format your answer exactly as follows (no explanations, na numbering):
Figure A6: The prompt used for summarizing the captions of optimal image groups with an LLM (Hidden CoT Prompting). The text in red represents the captions of the optimal image group, which depend on the target voxel and the number of optimal images used. The blue number specifies the required number of concepts, which was varied during the ablation study.
28

---

Published as a conference paper at ICLR 2026
Seeftence tarty
(sentence level)
Language
Cosine Sinilariy by
Generated caption
Model
sparts
SAaA COMMINI AEDI
Rufty white animal
showcasing the joytul
features
imaige
bumana and animals."
Meesured voxel activation
Inage stitad
captions
Capsionn
Model
Vose Pateeeoo
Image Generation Model
Viston
Model
Iiput Images
sperrs
skipp
toatures
Predicted voxel activation (image level)
Generated vaxel image
Imsqae Sirraabyy
Vision Model
Sinilerity
image
Figure A7: Overview of evaluation methods. (a) We first obtain text embeddings for both the generated voxel captions (from the language model) and the NSD image captions. We then compute the cosine similarity between the voxel caption embeddings and each NSD caption embedding to derive a rough prediction of voxel activity. Finally, we evaluate text-level prediction performance by calculating Spearman's 's rank correlation coefficient between these predicted values and the actual voxel responses. (b) We generate voxel images by visualizing voxel captions with an image generation model and, using the same vision model, compute vision-based embeddings for both the generated voxel images and the NSD image stimuli. As in (a), we compute the cosine similarity between voxel- image embeddings and NSD image embeddings and use Spearman's 's rank correlation coefficient to evaluate image-level prediction performance.
29

---

Published as a conference paper at ICLR 2026
Sentence level
Prediction Accuracy (p)
Prediction Accuracy (p)
BrainSCUBA
.0
LaVCa (Ours)
LaVCa (Ours)
LaVCa vs. BrainSCUBA
subj01
WWFA-1
subj02
subjo5
subj07
Figure A8: Mapping of brain activity prediction accuracy at the sentence level for LaVCa (left) and a comparison of brain activity prediction accuracy at the sentence level between LaVCa and BrainSCUBA (right) the flatmap for all subjects. The white outlines indicate Visual Word Form Area (VWFA-I) 1) and Occipital Word Form Area (OWFA), which are ranked among the top two Words-category ROIs based on the mean number of voxels across subjects.
30

---

Published as a conference paper at ICLR 2026
Image level
Prediction Accuracy
BrainSCUBA
Prediction Accuracy (p)
.0
.5
.5
(Ours)
LaVCa (Ours)
LaVCa vs. BrainSCUBA
subj01
subjo2
subj05
subj07
Figure A9; Mapping of brain activity prediction accuracy at the image level for LaVCa (left) and comparison of brain activity prediction accuracy at the image level between LaVCa and BrainSCUBA (right) onto the flatmap for all subjects.
31

---

Published as a conference paper at ICLR 2026
Table A8: Comparison of sentence-level brain activity prediction performance (all subjects). "Top-N voxels" refers to the voxels with top-N prediction performance in the training data. Each cell shows the mean & standard deviation of prediction performance on the test data. Two additional columns indicate the parammeer setting of LaVCa variants.
Top1000 voxels
Model
keywords
Sentence Composer
subj01
subj02
subj05
subj07
ShuMed
-0.010
0.094 + 0.312
0.145
0.331
0.265
BruinSCUBA
0.291 0.049
0.347 0.062
0.378
0.068
0.267 0.055
LaVCa (Ours)
X
0.300 + 0.054
0.352 = 0.057
0.393
0.059
0.286 0.056
LaVCa (Ours)
/
0.338 0.051
0.392 0.057
0.061 0.420
0.320 0.060
Top3000 voxels
Model
keywords
Sentence Compuser
subj01
subj02
subj05
subj07
Shuffled
0.000 0.228
0.059 0.255
0.099
0.274
0.004 0.205
BrainSCUBA
0.237 0.057
0.284 a: 0.067
0.305 0.077
0.212 0.061
LaVCa (Ours)
1
X
0.240 0.062
0.288 + 0.068
0.317 = 0.077
0.221 0.067
LaVCa (Ours)
5
0,280 = 0.059
0.325 # 0.068
0.349 = 0.075
0.253 0.069
Top5000 voxels
Model
keywords
Sentence Composer
subj01
subj02
subj05
subj07
Shufled
0.007 +: 0.199
0.058 + 0.223
0.067
0.243
0.009 0.175
BrainSCUAA
0.207 0.062
0,251 0.071
0.264
0.084
0.182 0.065
LaVCa (Ours)
1
X
0.205 + 0.068
0.250 = 0.075
0.272 = 0.086
0.186 0.072
LaVCa (Ours)
5
/
0.246 # 0.066
0.287
0.075
0.306 0.084
0.218 0.073
Top 10000 voxels
Model
keywords
Sentence Composer
subj01
subj02
subj05
subj07
Shuffled
0.008 0.157
0.039 0.178
0.051
0.192
0.012
BrainSCUBA
0.159 0.071
0.195 4 0.081
0.199 0.095
0.134 0.072
LaVCa (Ours)
1
X
0.154 = 0.076
0.190 : 0.086
0.199 = 0.101
0.132 0.080
LaVCa (Ours)
5
/
0.191 = 0.077
0.227 : 0.086
0.237 = 0.098
0.163 0.081
32

---

Published as a conference paper at ICLR 2026
Table A9: Comparison of image-level brain activity prediction performance (all subjects). "Top-N voxels" refers to the voxels with top-N prediction performance in the training data. Values are mean + standard deviation on the test data.
Top1000 voxels
Model
# keywords
Sentence Composer
subj01
subj02
subj0S
subj07
Shuffled
0.022 + 0.235
0.048
0.254
0.104 + 0.273
0.036 0.230
BrainSCUBA
0.278 + 0.056
0.322 0.052
0.357 # 0.057
0.262 0.061
LaVCa (Ours)
1
x
0.267 = 0.050
0.311 0.047
0.355 # 0.054
0.241 # 0.052
La (Ours)
0.314 + 0.059
0.347 0.053
0.379 : 0.054
0.289 0.060
Top3000 voxels
Model
# keywords
Sentence Composer
subj01
subj02
subj05
subj07
Shuffled
0.017 # 0.
0.059 4: 0.2
0.087 #: 0.228
0.007 0.174
BrainSCUBA
0.220 0,062
0.262 2 0.063
0.291 = 0.070
0.201 0.067
LaVCa (Ours)
x
0.213 + 0.058
0.255
0.058
0.292 + 0.067
0.187 0.061
La (Ours)
5
0.248 = 0.066
0.286 0.063
0.315 = 0.068
0.221 0.069
Top5000 voxels
Model
# keywords
Sentence Composer
suhj01
subj02
subj0S
subj07
Shuffled
0.017 0.163
0.052 0.185
0.066 0.204
0.009 0.148
BrainSCUBA
0.188 0.067
0.226 0,070
0.250 2 0.078
0.169 0.069
LaVCa (Ours)
1
X
0.182 : 0.063
0.221 0.066
0.252 : 0.077
0.158 0.064
LaVCa (Ours)
5
0.213 = 0.072
0.249 = 0.070
0.273 # 0.079
0.187 0.073
Top10000 voxels
Model
keywords
Sentence Composer
subj01
subj02
subj05
subj07
Shuftled
0.010 = 0.128
0.034
0.049 0.159
0.114
BruinSCUBA
0.139 0.073
0.170 0.081
0.188 0.090
0.122 0.073
LaVCa (Ours)
1
X
0.1 + 0.071
0.168 0.076
0.187 + 0.091
0.1 + 0.069
Va (Ours)
5
0.160 + 0.078
0.191 0,082
0.208 : 0.092
0.138 0,077
33

---

Published as a conference paper at ICLR 2026
subj01
subj02
subj05
subj07
330
15
tittionnn accursic Dentemec lavel
sediettn sccurscy Ope seee0orrc (cee))
Trediation Ieerrnnce
1:22
6:38
0:39
LuVCa BeandOUD
LaKia transouu
LeVCa 15A
Laice CreNCCUUAA
Number of words (eg-scala)
Number (log-scale)
Number words llog
Number worth llog scalel
Concat-50
A man wearing a beseball cap sticking his longue cut.
image
gill cupcakes placed her eyea, showcasing white and pink frosting with
smiley and
A person with sticker on their forehead and Union Jack their mouth.
A man with a beard and glasses hulcing an Orea paskage.
LaVCa-50
The image features cartoon pug wearing a bow tie and is surrounded by heart with banner that reads "Pugs and Kissos.".
food packaging features smiling person and cartoon character.
A man with a blue shirt is hold ng an iPod and smiling.
A man holding a box of Kallogg's Pop Tarts that has candles on it.
A child wearing a green and white striped shirt is smiling and holding mug that says "Happy Easter",
Accuracy: 0.235
A person holding a can of Dirtnday Cake Orea cookies.
Num. of words:
A ciose up a beg cat food called Whiskas.
A blue coffee mug with a red heart and a Daddy themed message next to an open creamer packet.
A man with glasses holcing a banana in front of his face.
A young woman with bracas is smiling while hoiding a largo wafflo in front her.
A cartzon bear holcing bar honey.
Three witers, A, B, and are displeyed on person's tongue.
Accuracy: 0.195
Num. words:
Figure A10: a Relationship between voxel caption prediction performance and word count (all subjects). The color of the plot corresponds to the lineage of each model. The numbers associated with LaVCa indicate the number of optimal images used for summarization, while the numbers associated with Concat represent the number of captions for concatenated optimal images. Error bars indicate the standard error. b Comparison of actual voxel captions between Concat-50 and LaVCa-50. Only portion of the captions is depicted for Concat-50.
34

---

Published as a conference paper at ICLR 2026
Table A10: Evaluation of the diversity three models. PCs (90% Var) means the number of principal components required to explain 90% variance of the text embeddings. For intra-voxel comparisons, the mean + standard deviation across subjects is presented. The inter-subject average (Average) is presented as the mean standard error.
Inter voxel
Intra- voxel
Lexical Vocab,
Semantic
Lexical
Semantic
Subject
Model
Variance
PC' (90% Var)
Vocab.
Length
Variance
BrainSCUBA
0.207=0.062
3400
6.21+1.27
1.46
0.016330.0025
subj01
Top-1 (Ours)
0.202=0.064
15384
0.0640
210
9.65=3.59
10.0s4.24
0.0194=0.0027
LaVC' (Ours)
0.24620.066
16477
0.0639
218
11.0a2.89
11.523.19
0.019820.0025
BrainSCUBA
0.251=0.071
3287
0.0591
133
6.17=1.32
6.27&1.51
0.0162:0.0026
suhj02
Top-1 (Qurs)
9.99=3.65
LaVCa (Ours)
0.287=0.075
17242
0.0639
218
11.3+3.64
11.8+3393
0.0198=0.0027
BrainSCUBA
0.26320.084
3043
0.0583
6.1821.37
6.2611.52
0.015920.0027
subjas
Top-1 (Ouns)
0.265:0.081
13485
0.0631
206
9.9923.68
10.4a4.34
0.0195=0.0028
LaVCa (Ours)
12.244.1
0.0199=0.0027
BrainCCUBA
0. 182=0.065
3042
0.0587
131
6.23e1.30
6.36tl 4.77
0.0163=0.0026
subj0?
Top-1 (Ours)
0.17920.066
12831
0.0632
203
10.123.51
10.624.11
0.019720.0026
LaVCa (Ours)
0.218=0.073
16508
0.0646
222
11.6a3.76
12.0a4.02
0.0202:0.0026
BrainSCUBA
0.226=0.019
3193a90
0.0588#0.0002
123+7.93
6.20=0.01
6.30+0.02
0.0162:0.0001
Average
Top-1 (Ours)
0.224=0.020
13959+555
0.063440.m0 00002
206+ .44
9.93=0.11 10
10.4+0. 1.2
0.01
LaVCa (Ours)
0.264:0.020
16922:252
0.0642+0.0002
219=1.00
11.420.175
11.920.149
0.0199=0.0001
35

---

Published as a conference paper at ICLR 2026
Table A11:
The average prediction accuracy for each subject and the inter-subject average prediction accuracy when captions were shuffled within the ROI (Shuffled) and when they were used as-is (Original). For each subject, the average prediction accuracy = standard deviation is depicted, while for the inter-subject average, the average prediction accuracy standard error is presented.
Body areas
Face areas
EBA
FBA-2
OFA
FFA-1
Shuffled
0.035+0.144 147
0.014+0. 128
0.031=0.067
0.017=0.113
subj01
Original
0.169+0.105
0.124+0.102
0.083+0.069
0.117+0.083
Shuffled
0.010=0.14 144
0.026#0. 109
0.036:0.007
0.024+0.097
subj02
Original
0.158+0.101
0.128+0.100
0.07940.066
0.105=0.008
Shufiled
0.001=00144 148
0,00740. 148
0.017#0.118
0.00940.
subj05
Original
0.152+0.111
0.149+0.114
0.120 100
0.112
Shuffled
0.028=0.135
0.025+0.096
0.02740.006
0.01340.T00
subj07
Original
0.149+0.106
0.099+0.099
0.097+0.099
0.108
Shuffled
0.018 008
0,018+0.00 0.0
0.028+0.004
0.016+0.005
Average
Original
0.157+0.005
0.125+0.010
0.095+00.009
0.111=0.003
Place areas
Word areas
OPA
PPA
OWFA
VWFA-1
Shuffled
0.080#0. 108
0.105+0. 107
0.015+0.057
0.05440.114
subj01
Original
0.163+0.093
0.172+0.099
0.055=0.048
0.147+0.088
Shuffled
0.11820. 139
0.17840. 147
0.037+0.071
0.031=0.135
subj02
Original
0.204+0.114
0.243+0.139
0.085+0.006
0.150+0.099
Shufiled
0.18420.140
0.21720.153
0.028 109
0,03920.148
subj05
Original
0.260+0.124
0.275+0.144 149
0.118+0.105
0.177+0.108
Shuffled
0.083=0.119
0.105+0.100 T08
0.020+0.070
0.01220.159
subj07
Original
0.175+0.096
0.163+0. 106
0.07940.066
0.157+0.118
Shuffled
0.116+0.024
0.151+0.028
0.025+00005
0.034 009
Average
Original
0.200+0.022
0.213+0.027
0.084+0.013
0.158+0.007
36

---

Published as a conference paper at ICLR 2026
0.5
subjo1
FBA-2 EBA
subj02
Body areas
subj05
subj07
subj01
subj02
subj05
0.4
subj07
subj01
OFA
subj02
Face areas
subj05
subj07
Cosine similarity
subj01
0.3
FFA-1
subj02
subj05
subj07
subj01
OPA
subj02
Place areas
subj05
subj07
0.2
subj01
PPA
subj02
subj05
subj07
VWFA-1 OWFA
subj01
subj02
0.1
Word areas
subj05
subj07
subj01
su
subj05
subj07
0.0
subj001
subjo2
Subijss
subbj77
subj01
subj02
subj05
subj07
subj01
subj02
subjos
subj07
subjo1
subj02
Subj05
subjo7
subj01
Subjj02
subj05
subj07
subj01
sub 05
Subj07
subj01
subj02
subj05
00
Subie
subj011
sabjns
subjoss
subj07
EBA
FBA-2
OFA
FFA-1
OPA
PPA
OWFA
VWFA-1
Body areas
Face areas
Place areas
Word areas
Figure A11: Cross-subjed similarity of voxel captions in ROIs. Each cell shows the mean cosine
similarity between sentence embeddings of voxel captions the two sets.
37

---

Published as a conference paper at ICLR 2026
Table A12: Voxel counts for each ROI, categorized by the number of clusters which each voxel belongs (# assigned clusters). Average rows show mean standard error across subjects.
Body
Face
Place
Word
assigned clusters
EBA
FBA-2
OFA
FFA-1
OPA
PPA
OWFA
VWFA-1
11
10
14
517
45
39
60
141
90
60
74
subj01
1083
126
130
151
465
288
136
257
904
369
165
263
377
82
53
84
325
226
83
149
50
19
11
10
87
46
16
23
517
137
49
44
182
140
39
1037
381
124
124
457
380
137
108
subj02
1081
424
147
112
337
207
134
22
1
39
7
29
19
13
subj05
1449
182
257
160
416
444
139
1446
181
248
111
450
379
147
1.7
829
65
139
41
218
177
73
92
214
26
1
34
11
5
34
8
303
69
26
45
193
118
40
subj07
158
88
88
298
1041
180
99
113
308
323
267
101
492
116
84
72
141
143
155
67
18
15
23
22
31
7
40:1-2
10::1
74:2
13:16
22:14
13:12
3=1
6::11
486-1-6
81 11:120
54
641:15
1741:11
130--17
50
49:10
1173-199
212-158
150-137
1311:16
433:-17
352.137
135-1
154
Average
233:644
1611:3
580-197
120.135
93118
62410
2251:38
166112
104.18
931-20
117::37
20:19
18:33
10::5
51::15
28:19
23=44
20::6
38

---

Published as a conference paper at ICLR 2026
OFA
Cluster
Cluster 2
Cluster 3
scar Bart passenger
spoon
S ggo
Clur
mnma imalffce
cortestt
ear
COOKy
Snoowooa de th
Copten ModTus
g Sroup
Fo
verson
EOCIOLE
aill
babyhuman
Staf
pinn Leppant
Werrket Sign p
poster
Ansach
S inhouette
decoration COS tume
chilo
man
man
Womann
Cluster 4
Cluster 5
Cluster
Car aass sert
em
advert
isement
cartoon
button icon bose
peop
e
S
stuffec
eee
000
photo
FADE eeselen
Lementt
plush
Intera acctinn
Kblue
alate
hair
Rood maddee
Cake sbArt
object c pr
product
EXDE
BRELOD
b
subj01
subj02
subj05
subj07
630
003
200
4.5
300
213
20
200
200
21
330
153
10
Cluseer
Clabbr
Cuxter
Figure A12: Subject-shared noun cluster analysis in the OFA. Word clouds and generated images for the top-5 most frrequent nouns in each subject-shared cluster. b Bar graphs showing the number of voxels assigned to each subject-shared cluster for individual subjects.
39

---

Published as a conference paper at ICLR 2026
PPA
Cluster 1
Cluster
Cluster 3
e emeeent
urban
SEO
group
bricge stat
Undoor EaT den Oom
building
sagn
computee ter
S
show
City
OLIC
Lannesss
W
item
SU
construct
SCIOY
serson
featurer
Cluster 4
Cluster
Cluster
1iving
g
spprtt T LOLLO
C
spa ipaces
stone
poo
tree
cass
titlee
table e
bathroom
window
furn
food
people
Water picture
Le
Fa Cappl L
Wooden
PRi
bo
helle volure
Kitchen
Staircas
Eattorm
é
subj01
subj02
subj05
subj07
400
131
Cluster
Cluster
Figure A13: Subject-shared noun cluster analysis the PPA. Word clouds and generated images for the top-5 most frrequett nouns in each subject-shared cluster. b Bar graphs showing the number of voxels assigned to each subject-shared cluster for individual subjects.
40

---

Published as a conference paper at ICLR 2026
Examples of OFA (1/2)
Top49 oothoated
Model
Voral cxption
Image
Acc (Tex--evel)
Acc. (imgge-lese))
Vocab size
Length
WSV verfance
organ
LavCa
decoration
9-154G
0.1094
14
0,02299
brand
(Durg)
Six offieren Tlxorr of Fring
chips are linc up side by sidc.
003317
BIan@CUBA
top of table.
hot
0.00001
0.01404
Topli actiorted
Model
Voxel caption
Vowel image
(Texs4evve
Length
A chiid weeting themeed ahitt, somezne brund 20
LaVCa (Dun))
person holding sign
B.03063
storehond PornT
Top (Dung)
brightly coloned and har Herrs on diplly.
O 0334S
0.096955
0.02171
ShanGCUBA
Worelin din 256665
Top49 Images
Voxel
Vocel
Apc. (Techlevel
(Image-iee
Ace.
Vecab.
Length
WAV variance
food paskloging foatures
LaVCa (Ound)
smilng person and carazon charadeer.
0U2399
Tap-1 (Cung)
A man weating besebial oap is straking ths tongue out
D. 1018
0.01982
mouth
0.17455
Topei activated
305512
Model
Vousl caption
Vocel image
(Toxi-level
(mage level)
size
Length
Gustom caricature loge ilustrutioo on fearurin the dignmum logo of
LavCe (Ours)
puspy eyeo complete wi cleck bick tooth and oie
0,163
0.1185
0.02005
Top-1 (Dun))
aheawa orllectioiet cupoalics with var ous medical
0.01900
therred decoral cone.
BOorSOBBA
n- los up of myy ouppakke on bod
0.1084
0.01547
Figure A14: Comparison of voxel captions and voxel images in the OFA voxels of subj02 (1/2).
41

---

Published as a conference paper at ICLR 2026
Examples of OFA (2/2)
Model
oaption
Vosel Image
(Texh-lvvo)
(mage-leveD
size
Length
varlance
Car boor handtngg baved good pokks dol porterned bieycle dossert.
B
0.1186
0.02518
Top- (Dam)
A white tike with block handiebars beaning apainst red baikk soel.
wall
bloyyle
0.01527
Weeriader 296464
Top49
Model
Vesel caption
Voosl image
Acc.
Acc:
The person pliningg animal mouth is person wealing glassee.
LaVCa (Dun))
Top-1 (Oum)
0.0410
0.04770
0.02047
person bussting
DOP77
Vosalin 277008
Tapil activated imagen
caption
Apc. (Tock-level
(Image
Acc.
Vocab.
Length
WIV variance
LavCe (Ouns)
decorative
0.0929
0.07104
0.01665
showes banaen
Top-1 (Dun))
plooed on brue unile
faoes drown them.
O 08960
0.1013
13
0.01781
BIdr
oniley
0.1057
0.01722
Topli activoted
Model
Voad caption
Vonel image
(Teckhleeel)
Length
WEV
Thai
decoration
LavCs (Dun)
scheme sared with sumples fom holddey theme mudke.
The mage foatureo a cowboy Inarred bartatey cale
Top-1 (Dam)
BorrSUUA
lubi sipped with calics Tap takees
0.01570
Figure A15: Comparison of voxel captions and voxel images in the OFA voxels of subj02 (2/2).
42

---

Published as a conference paper at ICLR 2026
Examples of PPA (1/2)
Model
Voxci osption
Voael Image
(mage-levee (Texhlevel)
size
Length
varlance
The mostrm almorphene.
8.327
0.1728
0.02066
Top-1 (Duma)
A rostaurtat led Fressoo noxt
1888
to arother medaurant calledt Kalatt
nocn building
0.01035
Weeriader 21766
Acc.
Acc:
Top49 aotivoted
Model
Vosel caption
Vooel Image
LaVGa (Dun))
shose sow insidw cake deplcting menneryuu
Top- (Oun)
gold
0.178
0.1185
0.01857
close stufied
O 1340
0.1154
0.01559
in 217847
Tapis activated imagen
caption
Apc. (Toct-ievel
(Imaee-lee Novel)
Acc.
Vocab.
Length
WIV variance
bathroon foture slices computer uter comgonent a wooden
LavCa (Ouns)
0.2785
0.2022
16
19
0.02198
cbject eovered with Dase paint and pattem
(Duns)
knnlo with various aisce up ows.
0 1778
0.1488
0.2299
BIdreOUBA
collage photos green
Voue
Model
Voac caption
Vonel image
(Teck-leee))
Length
WEV
Topei activated
This the pertect utten setting. with a brigge dock and veriele aboraee below.
LaVCs (Dun)
Tap-1 (Oum))
background.
BOrrSCUBA
rumber smail boots in
wader
0.01609
Figure A16: Comparison of voxel captions and voxel images in the PPA voxels of subj07 (1/2).
43

---

Published as a conference paper at ICLR 2026
Examples of PPA (2/2)
Model
Voxci osption
Voael Image
(Texhlevel)
(mage-leveD
size
Length
varlance
stainoase, view
8.273
0.2030
0.02104
Top- (Dui))
aciliial resert.
0,1006
water.
An nerial
0.01030
Weeriader 296ed1
Top49 adtivoted
Model
Vosel caption
Voosl Image
Acc.
Acc:
Longth
The peckedram informtuduure shows a train station and rillaay
LaVGa (Dun))
Top-l (Duug)
track that cayo Wuthortedd peroonel onyy"
O. 1963
0,1825
0.02029
street sign
O. 1ARR
0.2405
0.01280
in 200644
Tapii activoted imagen
caption
Apc. (Tock-level
(Image
Acc.
Vocab.
Length
WDV variance
LavCe (Ouns)
"Tee worden afrund inn, with adinn aschtectioe mimoss the zoo gate.
0.1727
0.00043
0.02812
Tap-1 (Cung)
eloments like ralling, steps,
Axden with
0 1400
0.1639
16
0.00 0.02167
lerterta, and naot.
BIdrEOUBA
shtting
Voue
Topli activated
Model
Voac ception
Vonel image
(Teck-leeel)
Length
WEV
The stiwior dee design highighhwd colortal righttine soome water feature.
LaVCs (Dun)
(Dus))
woodon
green
BorrSUUA
baffroom with sink, minoo, lolel andt bafttii..
C.01205
Figure A17: Comparison of voxel captions and voxel images in the PPA voxels of subj07 (2/2).
44

---

Published as a conference paper at ICLR 2026
EBA
subj01
subj02
subj05
subj07
Inter Culon on te
spo F. C
UOO ciiity
anteraction
Food
person
LeUTUe
berson
verb noun
gurond
DeFsoo
ACtiViL
OOTECE
SpOrT activity
Cnn
O
IN THE
sport
peop
e
pLaye
son
I
object DS RED
SPAT DO Centeeeer acttion
peop
500
nora
porfort P3P
OO
SPOFT
AO
weal
TOOK
MOW
P Watch Tide pose
Porter
wear
sannd
show
P
play
pEr
smile
pose
Spot
set
IN
S how MnICAR
ainteract act
stand
OFA
subj01
subj02
subj05
subj07
baby
an
person
peoble Lintefact Feetrrettn LION
GOE
aiimaa
Broup Kobba
verb noun
toy
peop cupcaker
pel
irnsonn Bo
Ltem
desserca
1000
WIRERTT TE
ODTECH
annmaa
GOUP
people
POOLO Homan Poga
pos
we
decorate
facecarry
boldd
wea faces how
Dose
e
S
show
spor
wear
make
sm show hol E
pose Teat for
smix Fe
hold
sappee eo
ture
smi
make!
smile
Feature
Chene
post
S
e
face
Show.wear
den
play
Figure A18: The UMAP projection of caption text across EBA and OFA for all subjects, visualized on a flatmap (top). A word cloud the 100 most frequent nouns these captions (middle), colored according their location in the UMAP space. word cloud of the 100 most frequent verbs (bottom).
45

---

Published as a conference paper at ICLR 2026
PPA
subj01
subj02
subj05
subj07
noun
scene
Luge
people
bui doing
paceconstructio
Kitchen
bectbahroonnurre
furnitur
tem
ture
bathroom
building
oomm
Dun
venic street
Lanng
kitchen
Toom
Streer
street
WLndOw
RUM
Foom
bu
ddin.
WI ndow
Plmment
watte trannnn
sink SE Ceneralie neateelee
verb
Featur
S
show
Recorate
now
aite Bice dinispa
ret retlect
ESM
Look
Wra S ShOW
inc Luderedic
Festurs
Stot ni Judeis 1ef
O
veel WOOK
Walk
se
ddecorate ataee
eature
set
Seleeshon show
ook
O ovel
LoOK
displer
Bitte frare
ee Teet ture
view..ree www.eetilleeeeeo luntnete
decorate
décor cone
OWFA
subj01
subj02
subj05
subj07
verb noun
Ob
HURO
person
SOLEAL
sign
la
000
descary
Lopcake
ser
gnitem
object temah
peop
wear D
people
chiLd
liveii
object ect
Ltemm
son
displey
people
telon l8Ct
bui luinging
Cos tume
irouu
BUrLding
child
animal
group
event
Warter
we
set
set
how
Gound
dow
Tace
smilafe eateetuo
take
decorate
mii
e
POS
e
show
sma
deRict
decorate
theme
O e
hold
hold wealo pose
heme
niemes
depict
how
Mist: show coold
eset Staeff eeeturuue
Figure A19: The UMAP projection of caption text across PPA and OWFA for all subjects, visualized on a flatmap (top). A word cloud of the 100 most frequent nouns these captions (middle), colored according their location in the UMAP space. word cloud of the 100 most frequent verbs (bottom).
46

---

Published as a conference paper at ICLR 2026
OFA
LaVCa
BrainSCUBA
subj01
subj02
Trem
Figure A20: Visualization of OFA captions for subj01 and subj02. For each subject, the captions' UMAP representations were mapped onto flatmap (top). The top 2 captions of each cluster in the UMAP space were visualized (bottom). The horizontal axis represents UMAP2, and the vertical axis represents UMAP2.
47

---

Published as a conference paper at ICLR 2026
OFA
LaVCa
BrainSCUBA
subj05
De geouge
subj07
Aigart Plan
Figure A21: Visualization of OFA captions for subj05 and subj07. The captions' UMAP representa- tions were mapped onto flatmap (top) for each subject. The top captions of each cluster in the UMAP space were visualized (bottom). The horizontal axis represents UMAP2, and the vertical axis represents UMAP2.
48

---

Published as a conference paper at ICLR 2026
PPA
LaVCa
BrainSCUBA
subj01
"Sheiing decpen
subj(2
Telle aiiee
Figure A22: Visualization of PPA captions for subj01 and subj02. The captions' UMAP representa- tions were mapped onto flatmap (top) for each subject. The top captions of each cluster in the UMAP space were visualized (bottom). The horizontal axis represents UMAP2, and the vertical axis represents UMAP2.
49

---

Published as a conference paper at ICLR 2026
PPA
LaVCa
BrainSCUBA
subj05
subj07
Figure A23; Visualization of PPA captions for subj05 and subj07. The captions' UMAP representa- tions were mapped onto flatmap (top) for each subject. The top captions of each cluster in the UMAP space were visualized (bottom). The horizontal axis represents UMAP2, and the vertical axis represents UMAP2.
50


Reviews:
[{"review_id": "jACgFa5LYS", "reviewer": "Reviewer_Bq4L", "summary": "This paper investigates the selectivity of voxels in the human visual cortex by generating a descriptive caption for each voxel. \nTo briefly summarize what the authors do:\n1. First they build voxel-wise encoding models using embeddings from CLIP\n2. They identify the \"optimal images\" that most strongly activate a specific voxel.\n3. Descriptive captions for these images are generated with MiniCPM-V.\n4. Using GPT-4o to extract keywords from the captions and composes them into a final, concise summary\n\nThe authors quantify the accuracy of their method in two ways:\n1. They use Sentence-BERT to create text embeddings for each generated voxel caption and for the captions of all images in the test set (NSD). For a given voxel, they calculate the cosine similarity between its caption's embedding and every test image caption's embedding. This similarity score is treated as the predicted brain activity. They compare this predicted activity vector to the voxel's actual measured activity using Spearman's rank correlation. A higher correlation indicates a more accurate caption.\n\n2. A similar eval where the voxel captions are turned into images using FLUX.\n\nThe authors find that their method is more accurate than BrainSCUBA.", "questions": "1. Line 195, do the authors convert the image embeddings to unit norm prior to constructing the encoder model?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The problem is a very interesting one, and understanding the selectivity distribution across the visual cortex can be of medical and scientific interest.\n\nThe approach is simple and consists of modular components that could be swapped out with future improvement in model quality.", "weaknesses": "1. It doesn't really make sense to me why the authors utilize an encoder to rank images in the first step. Each voxel has ground truth most activating images (from the fMRI beta weights). So it seems like this step is totally unnecessary and would degrade the model by replace real data rank with a predicted rank.\n\n2. The keyword extraction stage (Figure 3d) seems unnecessary. It is not clear that voxels would only reason to entities in an image, compared to actions or adjectives. \n\n3. The in-text citation format across the paper is very problematic. For example Line 135, the citations should not be in-text, and instead they should be in a parenthetical.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "rubDMtVjg3", "reviewer": "Reviewer_CUrs", "summary": "LaVCa introduces a data-driven approach for describing the selectivity of individual human brain voxels in the visual cortex using captions generated by large language models (LLMs). The proposed method trains encoding models for fMRI activity in response to images, isolates the most selective images for each voxel, uses a multimodal LLM for captioning, and finally composes concise keyword-driven voxel captions with an LLM and a keywords-to-sentence model. \n\nCompared to prior methods, the proposed method provides richer, more interpretable, and accurate natural-language descriptions of what each voxel encodes, revealing greater diversity and fine-grained functional specialization in the visual cortex.", "questions": "The following are some questions/suggestions for authors:\n\n- Can the pipeline be expanded to provide hierarchical or compositional captions, reflecting not just multiple keywords but structured relationships (object-action or scene-context)?\n\n- How does LaVCa generalize to multimodal responses, including voxels sensitive to auditory or language stimuli? Are the methods readily adaptable, or are critical modifications needed?\n\n- How reproducible are the identified semantic clusters across large populations? Do the same diversity patterns emerge in different datasets or subject groups?\n\n- Could direct behavioral validation (e.g., relating captioned selectivity to subject perception, imagery, or recall tasks) link voxel captions to cognition and perceptual experience more strongly?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "The strengths of this paper lie in its creative use of LLMs to generate natural-language captions that accurately describe voxel-level visual selectivity in the human cortex, surpassing prior methods in both interpretability and diversity of representations. \n\nThe approach is robust across benchmarks, clearly demonstrates broader and finer-grained conceptual tuning in visual areas, and is built with modularity and reproducibility in mind, thereby enhancing its impact on both neuroscience and neuroAI.", "weaknesses": "- While the captions are more diverse, the method often omits very local, fine-grained details in face-selective or highly specialized voxels, likely a result of summary steps and current limitations of captioning models?\n\n- Some hyperparameters (e.g., keywords and image set size) influence accuracy, and there is limited exploration of more structured or hierarchical compositional strategies for capturing multi-concept or multimodal selectivity.​\n\n- Benchmark comparisons focus primarily on accuracy and diversity, but lack direct behavioral validation. How these captions relate to actual perceptual or cognitive phenomena in human subjects could be better clarified.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "wOpU2ebufR", "reviewer": "Reviewer_7nH9", "summary": "This paper proposes a novel LLM-based pipeline for fMRI interpretation at the voxel level. The authors leverage the natural scenes dataset, and use the learned encoding weights from a VLM on NSD to predict fMRI responses to a novel set of images. They select the image set that produces the highest predicted response for each voxel and use LLM keyword extractors / sentence composers to provide rich and interpretable descriptions for each voxel.", "questions": "How do the voxel interpretations generated by keyword extractor / sentence composer directly on the captions (or an even simpler summary statistic of the captions) compare to the interpretations generated by LaVCa?\nWhat are the scores of the encoding model? How does the choice of VLM encoding model affect the captions / how can we assess that this backbone of LaVCa is accurate?\nHow can this method be extended to other datasets? What scale / diversity of sitmuli is needed?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The method leverages state-of-the-art AI methods to improve not only prediction but also interpretability of fMRI data\n\nThe model generalizes fMRI responses to held out images to generate rich descriptions of each voxel\n\nThe main figures replicate prior results of known neural tuning and may be a source of hypothesis generation for future studies.\n\nThe paper is well written with clear visuals", "weaknesses": "The main weakness is that it is unclear what the advantage of LaVCa is relative to using the direct fMRI responses from NSD. Given the large set of images shown in that fMRI dataset, which were drawn from MS-Coco, it seems possible to find the NSD image (or set of images) that drives the highest response in each voxel and do the same LLM extraction / sentence composition on the captions for those images. (This could be done in a cross-validated / encoding framework if desired.) In most of the paper, these captions are treated as the “ground truth” and it’s unclear what the advantage is for the first part of the voxel-preference pipeline. This major limitation significantly tempers claims about prediction accuracy (see below as well) and interpretability.\n\nWhile the paper is well written overall, the notion of prediction accuracy as cosyne similarity between the generated sentence or image and the original caption/image is not typical and not always clearly explained. For example, at a quick look Figs 2 and 4 could be interpreted as the more traditional encoding accuracy score (correlation between predicted and actual fMRI). More generally, this notion of prediction accuracy is extremely complex as it involves many steps of generative AI, removing it from the original data.\n\nAs a small point, the order of Fig 2 and FIg 3 should swapped.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "vogtNMKHrG", "reviewer": "Reviewer_NLFu", "summary": "This paper introduces LaVCa, a novel framework that leverages large language models (LLMs) to generate natural-language captions describing the selectivity of individual fMRI voxels in the human brain cortex. Although the proposed framework appears complex, the only trainable component is a standard voxel-wise ridge regression model, while all other modules remain frozen. Consequently, the methodological novelty primarily lies in the system design rather than in model learning or representation innovation.", "questions": "* Could the authors clarify whether the optimal image selection step could induce semantic biases that might affect voxel interpretations?\n\n* The manuscript reports comparison results with only one method; it is unclear whether other more recent or advanced methods could be included for comparison.\n\n* In Figure A3, how does a single voxel generate multiple captions? Is it based on different activation levels? The horizontal axis title in the upper right should be \"caption.\"", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "* The paper proposes a unique and well-motivated pipeline that applies LLMs to the problem of voxel-wise captioning. Overall, the paper is very well written, clearly organized, and easy to follow.\n\n* The captions generated by LaVCa quantitatively capture more detailed properties than the existing method. Both inter-voxel and intra-voxel analyses are thorough and supported by quantitative evidence.", "weaknesses": "* The approach relies heavily on the output quality of LLMs. While the pipeline is appealing, the lack of systematic ablation across different LLMs, prompts, or hyper-parameter settings limits the generalizability of the results.\n\n* The paper does not investigate how the choice of the external image dataset affects performance. All experiments rely on the OpenImages dataset.\n\n* In BrainSCUBA, caption generation relies on the nearest neighbor search operation. LaVCa also uses a similar operation, but with an external dataset.\n\n* LaVCa uses LLMs to explain voxel responses in natural language. Several recent works have explored generating natural-language descriptions directly from brain activity—although focusing on decoding rather than encoding, they share similar techniques, namely the alignment between model and brain representations. Despite this fundamental distinction, the current manuscript does not explicitly discuss how LaVCa complements or differs from existing brain-to-text work such as [1–2]. Without such discussion, readers may conflate LaVCa with decoding frameworks.\n\n  [1] Bridging the Gap between Brain and Machine in Interpreting Visual Semantics: Towards Self-adaptive Brain-to-Text Decoding, ICCV 2025.\n\n  [2] Mindgpt: Interpreting what you see with non-invasive brain recordings, IEEE TIP 2025.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)