TRIBE: TRImodal Brain Encoder for whole-brain fMRI response prediction
Authors: Stéphane d'Ascoli
Jérémy Rapin
Yohann Benchetrit
Hubert Banville
Jean-Remi King
Venue: ICLR 2026 Poster (2026)
Keywords: brain
encoding
multimodal
Primary area: applications to neuroscience & cognitive science
TL;DR: We introduce a deep learning model to predict brain responses to videos

Abstract: Historically, neuroscience has progressed by fragmenting into specialized domains, each focusing on isolated modalities, tasks, or brain regions. While fruitful, this approach hinders the development of a unified model of cognition. Here, we introduce TRIBE, the first deep neural network trained to predict brain responses to stimuli across multiple modalities, cortical areas and individuals. By combining the pretrained representations of text, audio and video foundational models and handling their time-evolving nature with a transformer, our model can precisely model the spatial and temporal fMRI responses to videos, achieving the first place in the Algonauts 2025 brain encoding competition with a significant margin over competitors. Ablations show that while unimodal models can reliably predict their corresponding cortical networks (e.g. visual or auditory networks), they are systematically outperformed by our multimodal model in high-level associative cortices. Currently applied to perception and comprehension, our approach paves the way towards building an integrative model of representations in the human brain. Our code is available at \url{https://anonymous.4open.science/r/algonauts-2025-C63E}.

Published as a conference paper at ICLR 2026
TRIBE: TRIMODAL BRAIN ENCODER FOR WHOLE-BRAIN FMRI RESPONSE PREDICTION
Stéphane d'Ascoli
Jérémy Rapin
Yohann Benchetrit
Meta AI
Meta Al
Meta AI
sdascol i @meta.
j jrapi wweetatte...mmmmmmm com
ybenchetr www.ittttta.....mmm
Hubert Banville
Jean-Rémi King
Meta AI
Meta Al
hubert jbmerttatemet.comm
jeanr wwwrremmmmeettt.
ABSTRACT
Historically, neuroscience has progressed by fragmenting into specialized do- mains, each focusing on isolated modalities, tasks, brain regions. While fruit- ful, this approach hinders the development of unified model of cognition. In this technical report, we introduce TRIBE, the first deep neural network trained to predict brain responses to stimuli across multiple modalities, cortical areas and individuals. By combining the pretrained representations of text, audio and video foundational models and handling their time-evolving nature with trans- former, our model can precisely model the spatial and temporal fMRI responses to videos, achieving the first place in the Algonauts 2025 brain encoding com- petition with a significant margin over competitors. Ablations show that while unimodal models can reliably predict their corresponding cortical networks (e.g. visual or auditory networks), they are systematically outperformed by our mul- timodal model in high-level associative cortices. Currently applied to percep- tion and comprehension, our approach paves the way towards building an inte- grative model of representations in the human brain. Our code is available http://twb..comccccccooooreeeeee wwwpaarrr//aaoaaataaaaaa algonauts 2025
INTRODUCTION
Motivation. Progress in neuroscience has historically derived from increasing specialization into cognitive tasks and brain areas. In the domain of vision, for instance, research focused on specialized cortical areas and their associated tasks, such motion perception in V5 (Shadlen & Newsome| 2001), face recognition the fusiform gyrus (Kanwisher & Yovel 2006), or the visual processing of written language the visual word form area (Dehaene Cohen (011). While this divide-and-conquu approach has undeniably yielded deep insights into the brain't 's mechanisms of cognition, it has led to fragmented scientific landscape: How neuronal assemblies together construct and globally broadcast unified representation of the perceived world remains limited to coarse conceptual models (Mashour et al. 2020).
The fast progress in AI in the domains of language (Brown et al. 2020 Grattatiori et al. 2024). image (Oquab et al.) 2023), audio (Baevski al.) 2020) Chung et al. [021] and video (Tong et al. 2022 Assran et al.) (025) may help resolve this fragmentation challenge. Indeed, the representations learnt by these AI models have been shown to - at least partially- - align with those of the brain (Huth)
ct al. 2016: Schrimpf et al. 2018 Caucheteux & King 2022). Motivated by this unexpected alignment, several teams have built encoding models to predict brain responses to natural stimuli from the activations of neural networks in response to images (Yang al. [023 Adeli al. 2023
Nguyen et al. 2023). speech (Millet et al., 2022) and text (Toneva & Wehbe 2019). However, these encoding models are currently limited in three critical ways.

---

Published as a conference paper at ICLR 2026
0.85
0.81
0.7.9
p-0.7
0.78
Figure 1: TRIBE predicts brain responses to videos across diverse regions. For eight brain parcels color-coded in the brain images, report the BOLD response of the first participant to the first minutes of held-out movie in solid lines and our model's predictions dashed lines, with the Pearson correlation of the two curves reported the left.
First, linearity: existing encoding approaches typically rely on ridge regression to map the AI model representations onto those of the brain. This assumes that these two sets of representations are linearly equivalent likely incorrect assumption (Linsley et al. 2025).
Second, subject-specificity: due to large variability in brain responses from one subject to another, existing encoding approaches typically train separate model for each subject, which prevents them from leveraging the similarities between brains.
Third, unimodality: most existing encoding approaches predict brain responses from unimodal stimuli, which makes them incapable of capturing how the brain integrates information from multiple modalities (Hu & Mohsenzadeh] 2025). This is particularly limiting as it has been shown that cross-modal interactions occur not only in specific multisensory areas (Gao al. 2023 Beauchamp 2005), but also in primary sensory areas (Driver & Noesselt, 2008 Stein & Stantord, 2008).
Contribution In this work, we introduce TRIBE, a novel deep learning pipeline to predict the (MRI brain responses of participants watching videos from the corresponding images, audio and transcript. This approach addresses the three limitations outlined above: our model learns how to capture the dynamical integration of modalities in an end-to-end manner across the whole brain, and from multiple subjects.
Our model achieves state-of- ff-thhe-n results, reaching the first place out of 267 teams in the Algonauts 2025 competition on multimodal brain encoding. We observe that the benefit of multimodality highest in associative cortices, and demonstrate via ablation analyses the importance of the multimodal, multisubject and nonlinear nature of TRIBE.
Related work While there has been recent research deep learning for multimodal brain decod- ing (Dahan et al. 2025) [Scotti et al., 2024:) Xia et al. [2024] Zhou et al., (024) Kong et al.) (024).. there currently exists equivalent for brain encoding. Some recent works suggest to train recurrent
2

---

Published as a conference paper at ICLR 2026
models to predict brain responses from frozen visual linguistic features (Gill & Van Gerven @017][hhhaal et al. 2021), or fine-tune existing pretrained models using the brain encoding objective. While these relax the linearity assumption, they restricted to a single sensory modality.
Conversely, a few recent studies have built encoding models on top Assion-lanuuee transformers,
demonstrating gains compared unimodal transformers (Dong & Toneva 2023 Oota al. 2022
Doerig et al. 2022 Wang et al. 2022 Tang et al. 2023). However, these works rely solely on linear mappings model brain responses from the activations of the multimodal transformers. We believe this can be suboptimal for two reasons. First, multimodal transformers are still relatively new: at rare exceptions (Jaegle et al., 2021] Srivastava & Sharma 2024] Abdin al., 2024). they often only integrate static images and text (audio and video being significantly more compute-intents tend to lag behind the performance of unimodal transformers. Second, and more fundamentally. the way these models integrate information across modalities may be very different from how the human brain does such multimodal integration. ideal encoding pipeline should thus learn how to best combine different modalities.
and
Modality dropout
Dynamical embeddings (f=2Hz)
Ground truth
Rachel
3007
Llama 3B
Text
-072
Hi Rachel
FFN
Loss: MSE
Metric:
Pearson
Transformer Encoder
Dudio =1024
Wav2Vec2- Bert
Audio FFN
Predictions
video 11408
Subject Layer
Video FFN
Pooling (f=1/TR)
Figure 2: Visual summary of our method.
2 METHODS
2.1 OVERVIEW
Task Our objective is to predict the brain activity of participants watching videos. This is framed as regression task where the targets the bood--oygee-lleel-deppee at every repetition time (TR=1.49s) of 3T fMRI recording device, separated into 1,000 non- overlapping parcels fig- [ For this, take as input the video clip being viewed by the participant, as well as the corresponding audio file and transcript. From these, we extract time-series of high- dimensional embeddings from the intermediate layers of start-of-the three modalities of interest: text, audio and video, which we feed to our deep encoding model, as illustrated in fig.
(BOLD) signals detected
generative Al models along
Evaluation To assess performance, evaluate for each parcel the Pearson Correlation p between predicted and ground truth fMRI responses across all TRs of the evaluation set. We then average these values over all parcels. We also refer to this metric as the "encoding score" of the model. We hold out 10% of the recording sessions for the validation set, ensuring that the same videos are held out for each participant prevent any form of data leakage.
3

---

Published as a conference paper at ICLR 2026
2.2 DATASET
Data collection We train our encoding model on the Courtois NeuroMod dataset (St-Laurent et al., (023). This dataset consists of six human participants who watched the same naturalistic videos, namely the first six seasons of the popular TV series Friends as well as four movies: The Bourne Supremacy, Hidden Figures, The Wolf of Wall Street and Life (a BBC Nature documen- tary). This amounts to an unprecedently large recording volume of over 80 hours of fMRI per subject. In the present work, we focus on a subset of four subjects curated for the Algonauts 2025 competition (Gifford et al. 2024
Preprocessing We use the preprocessing pipeline provided by the Algonauts2025 competition. The whole-brain BOLD IMRI responses are preprocessed using IMRIprep (Esteban et al. 2019) and projected to the Montreal Neurological Institute 2002). Functional images are then parcellated by averaging voxel-wise BOLD signals within each of the 1,000 parcels of the Schaefer atlas (Schaefer et al. 2018) 8), yielding a single fMRI time series for each of the parcels. Finally, activations were z-scored per parcel across each session (corresponding to approximately 15 minutes of recording).
standard space (Brett
2.3 MODEL
Timed text embeddings We extract "timed" text embeddings from the timestamped transcripts of the videos. For each word w to embed, we prepend the preceding which we feed through 3lama-3.2-3B (Grattafiori et al.) 2024). For each intermediate layer I, we extract the token(s) overlapping with the word w and average them to obtain a contextualized word embedding of dimension D 3072. We then construct an evenly spaced grid at frequency f = 2 Hz, and for each time-bin т, we sum the embeddings of words which overlap with the bin. This allows to temporally align the text features with the audio and video features.
1, 024 words in the transcript,
Audio embeddings To obtain audio embeddings, we extract audio files from the videos, split them into 60-second chunks, then feed these through Wav2 Veav2Vec-err-22 (Chung et al. [021). We then resample the hidden representations of the latter from 50 Hz to f 2 Hz. For each intermediate layer I, this yields time series of embeddings of dimension Daudio 1024. Note that the resulting embeddings carry bidirectional information about both the past and future of the stimulus window, whereas text and video embeddings only contain information about the past.
Video embeddings For video embeddings, we again construct an evenly spaced grid at frequency f = 2 Hz, and for each bin of time, we feed 64 frames spanning the preceding 4 seconds to Video-JEPA gigantic (Assran et al., 2025). For each intermediate layer I, we compress the tensor of activations by averaging over all patch tokens, yielding a time series of embeddings of size Dviden = 1408. Note that this spatial averaging step was necessary to keep the size of the tensor manageable. However, it comes at the cost of discarding positional information, which we expect to deteriorate encoding performance low-level visual areas which exhibit retinotopic mapping (Wandell & Winawer 2011)
Combining the modalities For each of the three modalities m, the feature extraction described above leads to a time series of embeddings at f = 2z, with embeddings of shape [Lm, Dml, where 4n and Dm are the number of layers and dimensionality of the transformer of modality m. To compress these embeddings while retaining both low-level and high-level information, for each modality, we split the layers into groups, then average the tensor per group along the layer dimension, compressing to shape [T. D..)- As shown in fig. 7 for each modality, the embeddings coming from deeper layers of the corresponding foundational model yield better encoding performance, especially in associative cortices (fig. 8 and the best configuration was found to be L = 2 groups ranging from relative depths 0.5 to 0.75, and 0.75 to 1. We then concatenate the layers and feed the resulting vector through linear layer with shared output dimension D = 1024 followed by layer normalization. Finally, we concatenate the three modalities, leading to a time series of multimodal embeddings of shape 3 x 1024. This will be the input to our transformer encoder.
4

---

Published as a conference paper at ICLR 2026
Extracting windows From these time series of stimulus embeddings, extract windows of duration T = .N x TR, as well as the corresponding N target BOLD responses. Since the number of input tokens is N TR x f. at fixed memory budged, there is tradeoff between increasing the stimulus sampling frequency f and the window duration N. Through a grid search with frequencies ranging from 0.5 to 10 Hz fixed number of input tokens, we found f = 2 Hz and N 100 to yield the best results.
Transformer encoder We add to the stimulus embeddings learnable positional embeddings, then feed the result through Transformer encoder with 8 layers and 8 attention heads. This enables information to be exchanged between timesteps. At the output the transformer, we use an adaptive average pooling layer compress the sequence from length fT to N, yielding one embedding per TR.
Subject layer Brain responses to a given stimulus vary from one brain another. To enable training a single model across all subjects, we use a subject-conditional layer, following (Défossez) al. 2023), which selects different linear projection for each subject. The latter projects the output of the transformer to the 1, 1,000-dimensions target space. Note that all timesteps in the window are predicted simultaneously, which makes our model particularly compute-efiiee -fffciiete at inference.
Hemodynamine lag To avoid border effects due the hemodynamic lag between the stimuli and the corresponding brain responses, we offset targets by seconds relative to the inputs. Note that since the transformer has access to all timesteps, encoding of convolving the inputs by a temporal response function. Instead, the attention mechanism selects the most relevant timesteps: peak between 5 and 10 seconds relative to the current timestep, which is consistent with the expected hemodynamic response function.
do not follow the common practice in linear
analysis presented in fig. shows that the attention weights
2.4 TRAINING DETAILS
One desirable property of a multimodal encoding model is its ability to provide meaningful predictions in the absence of one or several modalities, for example for silent movie or a podcast. To encourage this behaviour, while at the same time avoiding excessive reliance on one modality, we introduce modality dropout: during training, we randomly mask off each modality by zeroing out the corresponding input tensor with a probability p. ensuring that at least one modality is left unmasked.
Modality dropout
Optimization We train our model for up to 15 epochs with the AdamW optimizer (Loshchiloy & Hutter, 2017) using batch size of 16. The learning rate is warmed up linearly to 10 over the first 10% of steps. then decayed following cosine learning rate schedule. We use carly stopping based on the validation Pearson score computed on held-out set. To improve generalization. we use stochastic weight averaging (Izmailov et al., 2018), which involves averaging model weights obtained at the end of each epoch, once the validation metrics are near their plateau.
Ensembling To further improve generalization, ensemble the predictions of M 1000 models, whose initialization and shuffling seeds are all different. To strengthen ensemble diversity, for each model, we sample separately, we compute the encoding score of all models on the validation set, then compute a softmax distribution over models with temperature 0.3, which will determine the weight assigned to each model for this given parcel.
of hyperparameters uniformly in the grid specified in table For each parcel
Implementation details We extract stimuli features from pretrained language, audio and video models available the Huggi wooggingFast ace platform (Jain 2022) and cache them as Numpy memmap arrays (Harris et al., 2020) for fast loading during the training of our encoding model. These models respectively contain 3B, 600M and 700M parameters, and feature extraction is completed in 24 hours on 128 V 100 GPUs with 32GB of VRAM. The TRIBE model itself contains contains 980M trainable parameters, and the full training loop lasts 24 hours on single such GPU. We the transformer
5

---

Published as a conference paper at ICLR 2026
implementation from the x-tzansformers package
We list the licenses of the assets used in
this work section|]
Rank
Mean score
Subject 1
Subject
Subject
Subject 5
I (Ours)
0.2146 + 0.0312
0.2381
0.2105
0.2377
0.1720
2
0.209 0.0283
0.2353
0.2046
0.2268
0.1718
3
0.2094 + 0.0215
0.2233
0.2072
0.2271
0.1798
0.2085 0.0267
0.2295
0.2003
0.2300
0.1743
5
0.2055 + 0.0291
0.2306
0.2010
0.2240
0.1662
Table Our model achieves first place in the Algonauts 2025 leaderboard. We report the mean
score + standard deviation for the top five out of 267 teams.
OOD
Movie
Mean
Subject
Subject
Subject 3
Subject
X
Friends Season
0.3195
0.0289
0.3419
0.3239
0.3346
0.2775
Pulp Fiction
0.2604
0.0137
0.2765
0.2611
0.2431
0.2610
V
Princess Mononoke
0.2449
0.0572
0.2816
0.2507
0,2851
0.1623
/ Passe-partout
0.2323 = 0.0525
0.2763
0.2587
0.2370
0.1573
/ World of Tomorrow
0.1924 0.0323
0.2210
0.1606
0,2196
0.1686
/ Planet Earth
0.1886 = 0.0380
0.1483
0.2029
0.2331
0.1699
/ Charlie Chaplin
0.1686 = 0.0551
0.2249
0.1289
0,2080
0.1128
Table 2: Our model generalizes highly out-of-distribution movies. We report the mean standard deviation for the held-out datasets of the Algonauts 2025 competition.
3 RESULTS
3.1 ALGONAUTS 2025 COMPETITION RESULTS
Our model achieves the first place out of 267 teams in the Algonauts 2025 competition multimodal brain encoding. As shown in table we outperform competitors by a substantial margin: the gap between our model and the runner-up is larger than between the runner-up and the fifth.
We display the results across the various held-out datasets table the dittiitribution setting of the first phase of the competition (Friends season 7), our model achieves a mean score of 0.3195. Unsurprisingly, when tested in the ant-of-distribution setting of the second phase of the competition, performance is lower, with average of 0.2146. Remarkably, our model achieves robust scores even the extreme ouitof-distribution setting of cartoons (0. 1924 for World of Tomorrow) wildlife documentaries (0. 1886 for Planet Earth), and silent black-and movies 10.1686 for Charlie Chaplin), where language processing cortices are very poorly encoded (see fig. 10).
3.2 WHOLE-BRAIN ACCURACY
How the prediction accuracy distributed spatially? We report the Pearson scores across the cortical surface in fig. 3 The distribution is highly non-uniform (panel a), and peaks in the temporal and occipital areas of the brain associated with auditory, language and visual processing (panel b). To assess which brain parcels predicted significantly better than chance, permutation test (1000 permutations) across the 164 recording sessions of the held-out validation set. Remarkably, all 1000 parcels are significantly predicted (g(FDR) 10"3)
biock-wise

---

Published as a conference paper at ICLR 2026
1.0
0.60
0.8
Pearson
0..40
0.20
0.6
0.00
0.4
0.80
Normalizz lldd
0.2
0.60
0.40
0.0
0.20
Pearson
Noise
Normalized Pearson
0.00
ceiling
(a) Distribution across parcels
(b) Cortical projection of the encoding scores
Figure 3: Our model predicts responses significantly across the whole brain.
(a) We report histogram of unnormalized and normalized encoding scores (see definition in eq. of the 1000 parcels, averaged across validation sessions. All 1000 parcels are predicted significantly above chance (f(DRR < 10-3, block-wise permutation test across sessions).
(b) We project the encoding scores averaged across subjects onto the fsaverage cortical surface.
3.3 NOISE CEILING ANALYSIS
To what extent are encoding errors due to unexplainable randomness rather than model suboptimality? Following common practice in the brain encoding literature, we estimate the noise ceiling by computing the inter-trial correlation peety of the BOLD responses to the movies Hidden Figures and Life, which were viewed twice by each participant. We then define the normalized Pearson correlation of our model by dividing it by that of an ideal encoding model (Schoppe et al. 2016):
2
Pnorm
Praax
Pmazz
V
1+ Puets
Our model achieves normalized Pearson correlation of 0.54 + 0.1 across all parcels (fig. : in other words, captures roughly half of explainable variance on average for the movies considered. The fairly small inter-quartile range reflects the fact that our model is rather consistent across brain areas. The highest values are achieved in the auditory and language processing cortices, where our model captures above 80% of explainable variance (fig. 3p).
3.4 THE BENEFIT OF MUL.IMODALIIYY
To what extent do the three modalities combined by TRIBE contribute to encoding performance? We address this question in fig. 4h. by assessing the encoding performance TRIBE retrained with various modalities ablated. When training encoding scores. We observe that the text modality achieves the lowest average encoding score with 0.22, followed by audio at 0.24 and video at 0.25. When combining any two modalities together, the encoding scores rise significantly compared to the unimodal models: the best bimodal model, obtained by combining text and video, achieves encoding three modalities together yields an additional boost, bringing the value to 0.31. This hints to the fact that each modality plays complementary role.
single modality, TRIBE achieves substantially lower
of 0.30. Finally, combining the
7

---

Published as a conference paper at ICLR 2026
T=text A=audio V=video
T-
0.05
A
0.05
0.03
0.00
A+V
T+V
-0.0%
A+T+V
0.10 0.15 0.25
-0.30
Validation Pearson
(a) Modality ablations
(b) Multimodal versus best unimodal
Figure 4: Multimodal encoders outperform unimodal encoders.
(a) We compare the encoding scores of encoders trained subset of modalities in the same conditions. Error bars denote s.e.m across subjects.
(b) For each parcel of the first subject, we compute the difference in encoding score between the multimodal encoder and the best out of the three unimodal encoders, then project the results onto the fsa ve rage cortical surface.
Score with audio only
Text
Audio
Video
video only
Score
V
Score with text only
(a) Highest scoring modalities
(b) Interplay between modalities
Figure 5: The modalities and their interplay map onto the expected brain areas.
(a) For each parcel of the first subject, we assess the encoding contribution of each modality by masking the two other modalities, and color-code according to the highest contribution.
(b) We color-code each parcel using an RGB encoding where red, green and blue intensities are determined by the encoding score of the model with text, audio and video unmasked, respectively. For readability, we limit ourselves to bimodal interactions by substracting the smallest of the three contributions. Red, blue and green areas correspond to unimodal areas well encoded by text, audio and video respectively, while magenta, yellow and cyan correspond to bimodal areas well encoded by text-video, text-audio and video-audio interactions respectively.
In which areas does multimodality yield the strongest gains? In fig. 4p. we compare for each parcel the encoding score of the multimodal encoder with that of the best of the three unimodal encoders. We observe that the multimodal encoder consistently outperforms the unimodal models, especially in associative areas such the prefrontal parieto-ocippio-tempoll cortices (up 30% increase in
8

---

Published as a conference paper at ICLR 2026
0.32
0.220
0.30
0.215
0.25
0.205
0.26
0.200
0.10
0.24
0.05
0.190
Ours
Without multisubject transformer
Without
10
10a
10a
102
10a
Number of sessions per subject
Number of context words
(a) Model ablations
(b) Number of sessions
(c) Context length
Figure 6: Ablations and scaling laws of our model.
(a) We compare the results of our model with those achieved without multi-subject training or without transformer model.
(b) We report the results obtained by the multimodal encoder as we increase the number of sessions used the training set.
(c) We report the results obtained by the text-only encoder increase the context length of the language model used to extract features (Llama-3.2-3BB,
In all panels, shaded regions indicate the s.e.m over the four subjects.
encoding score). Interestingly, the multimodal model performs less well than the vision-onk, -olly model the primary visual cortex, which highly specific to visual features.
Together, these results demonstrate that our multimodal encoder effectively captures interactions between modalities, which improves whole-brain decoding.
3.5 THE INTERPLAY BETWEEN MODALITIES
Which brain areas is dominated by unimodal or multimodal representations? To address this question, we single out modalities by probing our multimodal model with all modalities masked off except one (using the same procedure for the dropout described in section 2
In fig. 5h. we color each parcel of the cortex according to the modality that achieves the highest encoding score via this procedure. The three modalities each cover broad regions: audio predominates near the temporal gyrus, video predominates in the occipital cortex and parts of the parietal cortex, while the text feature, which presumably contains the most semantic information, predominates in large parts of the parietal and prefrontal lobes.
To study the interplay between modalities, we then overlay the contribution of the three modalities using RGB encoding where red, green and blue respectively represent the encoding scores achieved solely with text, audio and video (fig. 5p)). make hues more visible, we restrict our analyses to unimodal and bimodal interactions by substracting the smallest contribution among the three modalities. We observe interesting bimodal associations in some key areas: in particular, text+audio (yellow) in be observed in the superior temporal lobe and video+audio (cyan) can be observed in the ventral and dorsal visual cortices.
Overall, these observations provide insights how multisensory integration may occur in the human cortex.
3.6 ABLATIONS AND SCALING LAWS
fig. [6). we demonstrate the importance of using a transformer and a multi-subject training scheme: the encoding score drops from 0.31 to 0.29 when training separately for each subject, and down to 0.23 when removing the transformer. In fig. op. we show how the encoding score scales with the amount of recordings in the training set. We observe strong increasing trend which has not reached a plateau, line with recent work (Antonello et al.) (023). Finally, in fig. (b. we show that increasing the context length used for the language model words strongly enhances encoding performance,

---

Published as a conference paper at ICLR 2026
without any plateau very long context lengths of 1024 words. This confirms that TRIBE effectively captures high-level semantic features, far beyond the word and sentence level.
4 DISCUSSION
In this work, we make a step towards an integrative model of the brain during naturalistic perception by training encoding model on an unprecedently-lne -large fMRI dataset of participants watching videos. Crucially, our model is the first encoding pipeline which is simultaneously nonlinear, multisubject and multimodal: our ablations demonstrate the importance of these three aspects for encoding, especially in associative cortices. Our model achieves the first place in the Algonauts 2025 brain encoding competition, and scaling laws suggest that encoding performance increases systematically with the number of recordings, holding promise for further improvements with larger datasets.
Limitations There are several remaining limitations to our work. First, our approach currently operates coarse parcellation of brain areas reducing hundreds of thousands of voxels to 1,000 cortical parcels. This design choice, introduced by the Algonauts2025 challenge, likely increases the signal-to-non noise ratio by smoothing out the responses spatially, and certainly reduces computational cost, which is important for the whole-brain prediction setting considered here. However, this approach limits the spatial resolution of our model, which inherently prevents from capturing highly localized phenomena. Adapting our model for voxel-level prediction is an important avenue for future work. Second, our current approach is limited to fMRI data. Consequently, the precise temporal dynamics of neuronal activity, and the exact neural assemblies underlying these macroscopic signals, remain, here, unresolved. Applying a similar approach for electro- signals could prove invaluable in this regard. Third, while the volume of recording per participant in the study considered here particularly large, only four participants were included: extending and improving our results on larger pool of participants is an important next step. In particular, whether TRIBE can generalize to unseen subjects, either in a zero-shot or few-shot setting, is an important question for future work, which will likely require naturalistic datasets with larger pool of participants, such as that of the Human Connectome Project (Van Essen & Ugurbil 2012). Fourth, our model predicts responses deterministically from the perceptual inputs, at odds with the brain for example: in the absence of stimuli, the default mode network is expected to display a complex dynamical pattern (Raichle 2015), which cannot be captured by our model. Transitioning towards generative methods such as diffusion models would be necessary to capture such phenomena. Finally, the present approach remains limited to perception and comprehension. Behavior, memory and decisions are other important cognitive components to integrate into the present model.
puagneto-cneephalograhhh
Broader impact Building a model able to accurately predict human brain responses to complex and naturalistic conditions is important endeavour for neuroscience. Not only does this approach open the possibility of exploring cognitive abilities (e.g. theory of mind, humour) that are challenging to isolate with minimalist designs. but they will eventually be necessary evaluate increasingly complex models of cognition and intelligence. integrate the different sub-fields of neuroscience into single framework and to (2) develop in silico experimentation (Jain et al. 2024), where in vivo experiments could he complemented and guided by the predictions of brain encoder. While the exploration of this paradigm falls beyond the scope of this technical report, we believe that epistemology is ordered: interpretation and scientific insights are most legitimate if they derive from the model that makes the best prediction. In this regard, the first place achieved by TRIBE in the Algonauts 2025 competition gives scientific credit to our approach.
addition, our approach forges a path to (1)
10

---

Published as a conference paper at ICLR 2026
Ethics statement We aim for whole-braie encoding framework to chart a viable path toward foundational model of cognition - one that bridges the diverse avenues of neuroscientific research. We do however call for caution concerning the negative avenues of this line of research: brain response prediction could become a lucrative endeavour which may exploit our cognitive weaknesses, e.g. to amplify harmful consumption habits (Murphy al. 2008).
Reproducibility statement We provide fully runnable codebase to reproduce our results at the 2005--06666
following address: htttps://aaonnmmttt 4open. wwtienc//e/aaeaeeeeeeeeeeeeeeeee
REFERENCES
Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi. Piero Kauffmann, James R. Lee, Yin Tat Lec, Yuanzhi Li, Weishung Liu, Caio C. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang. Rachel Ward, Yue Wu, Dingli Yu, Cyril Zhang, doi: 10448550/rXx..22122.88889
and Yi Zhang. Phi-4 Technical Report. arXiv, December 2024.
Hossein Adeli, Sun Minni. and Nikolaus Kriegeskorte. Predicting brain activity using transform- ers. bioRxiv, 2023. doi:
URL (ntthtt::: //www. www.xii.ooooooooo....o...ooooo.e
Corttent/earll//222///////////
08 5517433
Richard Antonello, Aditya Vaidya, and Alexander Huth. Scaling laws for language encoding models in fmri. Advances in Neural Information Processing Systems, 36:21895-21999 2023.
Mido Assran, Adrien Bardes, David Fan, Quentin Garrido, Russell Howes, Matthew Muckley, Ammar Rizvi, Claire Roberts, Koustuv Sinha, Artem Zholus, al. V-jepa 2: Self-supervised video models enable understanding, prediction and planning. arXiv preprint 25Xiv:2506.09985,, 2025.
Alexei Baevski, Yuhao Zhou, Abdelrahman Mohamed, and Michael Auli. wav2vec 2.0: A framework self-supervised learning of speech representations. Advances in neural information processing systems, 33: 1444--224660 2020.
Michael S Beauchamp. See me, hear me, touch me: multisensory integration in lateral occipital- temporal cortex. Current opinion in neurobiology. (5(2):145-13,, 2005.
Matthew Brett. The mni brain and the talairach atlas.
www. mre-Mre-chu. cam. ac.
uk/hmaagin/mniispace.. Hmml, 2002.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 33:1877-1901, 2020.
Charlotte Caucheteux and Jean-Rémi King. Brains and algorithms partially converge in natural language processing. Communications biology, 5(1::334, 2022.
Omar Chehab, Alexandre Defossez, ChrisShritepphe Loiseau, Alexandre Gramfort, and King. Deep recurrent encoder: A scalable end-to-end network to model brain signals. arXiv
preprint arXiv:2 103 02339, 2021.
Yu-An Chung, Yu Zhang, Wei Han, Chung-Chems Chiu, James Qin, Ruoming Pang, and Yonghui Wu. Wev-bert: Combining contrastive learning and masked language modeling for sulf-supervised speech pre-training. In 202/ IEEE Automatic Speech Recognition and Understanding Workshop (ASRU), pp. 244-250 IEEE, 2021.
Simon Dahan, Gabriel Bénédict, Logan Zane John Williams, Yourong Guo, Daniel Rueckert, Robert Leech, and Emma Claire Robinson. Sim: Surface-based fmri analysis for inter-subject multimodal decoding from movie-watching experiments. In The Thirteenth International Conference on Learning Representations, 2025.
Alexandre Défossez, Charlotte Caucheteux, Jérémy Rapin, Ori Kabeli, and Rémi King. Decod- ing speech perception from non-invasive brain recordings. Nature Machine Intelligence, 5(10): 1097-1107, 2023.
11

---

Published as a conference paper at ICLR 2026
Stanislas Dehaene and Laurent Cohen. The unique role of the visual word form area in reading. Trends in cognitive sciences, 15(6):254-262, 262, 2011.
Adrien Doerig, Tim C Kietzmann, Emily Allen, Yihan Wu, Thomas Naselaris, Kendrick Kay, and Ian Charest. Semantic scene descriptions an objective of human vision. arXiv preprini MxXiv:220.111332 10, 2022.
Dota Tianai Dong and Mariya Toneva. Vision-language integration in multimodal video transformers (partially) aligns with the brain. arXiv preprint urXiv:2311.07766 2023.
Jon Driver and Toemme Noesselt. Multisensory interplay reveals crossmodal influences on 'sensory- specific apecific'brain brain regions, neural responses, and judgments. Neuron, 57(1): 2008
Oscar Esteban, Christopher J Markiewicz, Ross W Blair, Craig A Moodie, A Ilkay Isik, Asier Erramuzpe, James D Kent, Mathias Goncalves, Elizabeth DuPre, Madeleine Snyder, et al. fmriprep: robust preprocessing pipeline for functional mri. Nature methods, 16(1):111-11) 2019.
Chuanji Gao, Jessica Green, Xuan Yang, Sewon Oh, Jongwan Kim, and Svetlana V Shinkareva. Audiovisual integration in the human brain: a coordinate- basedd meta- meta-analysis. Cerebral Cortex, (3(9):5574-5584 2023.
Alessandro Gifford, Domenic Bersch, Marie St-Laurent, Basile Pinsard, Julie Boyle, Lune Bellec, Aude Oliva, Gemma Roig, and Radoslaw M Cichy. The algonauts project 2025 challenge: How the human brain makes sense of multimodal movies. arXiv preprint arXiv:2501.00504 2024.
Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The Ilama 3 herd of models. arXiv preprint arXiv:2407.2178. 2024.
Umut Güçlü and Marcel AJ Van Gerven. Modeling the dynamics of human brain activity with recurrent neural networks. Frontiers computational neuroscience, 11:7, 2017.
Charles Harris, K Jarrod Millman, Stéfan Van Der Walt, Ralf Gommers, Pauli Virtanen, David Cournapeau, Eric Wieser, Julian Taylor, Sehastian Berg, Nathaniel Smith, et al. Array program- ming with numpy. Nature, 585(7885)) 2020.
Yu Hu and Yalda Mohsenzadch. Neural processing of naturalistic audiovisual events in space and time. Communications Biology. 8(1): 10, 2025.
Alexander G Huth, Wendy A De Heer, Thomas L Griffiths, Frédéric E Theunissen, and Jack L Gallant. Natural speech reveals the semantic maps that tile human cerebral cortex. Nature, 532 (7600):453-455 2016.
Pavel Izmailov. Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson, Av- eraging weights leads to wider optima and better generalization. arXiv preprint 1eXivv/803.05400 2018.
Andrew Jaegle, Sebastian Borgeaud, Sean-Baptiste Alayrac, Carl Doersch, Catalin Ionescu, David Ding, Skanda Koppula, Daniel Zoran, Andrew Brock, Evan Shelhamer, Olivier Hénaff, Matthew M. Botvinick, Andrew Zisserman, Oriol Vinyals, and João Carreira. Perceiver IO: General Archi- tecture for Structured Inputs & Outputs. arXiv, July 2021. doi: 10448550urxxiv22007..79999
Shailee Jain, Vy A Vo, Leila Wehbe, and Alexander Huth. Computational language modeling and the promise of in silico experimentation. Neurobiology of Language, 5(1):80-106, 2024.
Shashank Mohan Jain. Hugging face. In Introduction to transformers for NLP: With the hugging face library and models to solve problems, pp. 51-67. Springer, 2022.
Nancy Kanwisher and Galit Yovel. The fusiform face area: a cortical region specialized for the perception of faces. Philosophical Transactions of the Royal Society B: Biological Sciences, 361 (1476):2109-22 -228, 2006.
Xiangtao Kong, Kexin Huang, Ping Li, and Lei Zhang. Toward generalizing visual brain decoding to unseen subjects. arXiv preprint arXiv:2410. 14445, 2024.
12

---

Published as a conference paper at ICLR 2026
Drew Linsley, Pinyuan Feng, and Thomas Serre. Can deep neural networks learn biological vision? arXiv preprint arXiv:2504. 16940, 2025.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint
1rXiv:1711.05101, 2017.
George A Mashour, Pieter Roelfsema, Jean-Pierre Changeux, and Stanislas Dehaene. Conscious processing and the global neuronal workspace hypothesis. Neuron, 105(5):776-7988 2020.
Juliette Millet, Charlotte Caucheteux, Yves Boubenee, Alexandre Gramfort, Ewan Dunbar, Christophe Pallier, Jean-Remi King, et al. Toward realistic model of speech processing in the brain with self-supervised learning. Advances in Neural Information Processing Systems, 35:33428-33443, 2022.
Emily R Murphy, Judy Illes, and Peter Reiner. Neuroethics of neuromarketing. Journal of Consumer Behaviour: An International Research Review, 7(4-5):293-30,, 2008.
Xuan-Bac Nguyen, Xudong Liu, Xin Li, and Khoa Luu. The algonauts project 2023 challenge: Uark-ualbany team solution. arXiv preprint wxxv:23388.0026. 2023.
Subba Reddy Oota, Jashn Arora, Vijay Rowtula, Manish Gupta, and Raju S Bapi. Visio-linguistic brain encoding. arXiv preprint arXiv:2204.0m 08261, 2022.
Maxime Oquab, Timothée Darcet, Théo Moutakanni, Huy Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez, Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby, et al. Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:230.007193, 2023.
Marcus E Raichle. The brain's 's default mode network. Annual review of neuroscience, 38(1):333444, 2015.
Alexander Schaefer, Ru Kong, Evan M Gordon, Timothy O Laumann, Xi-Nian Zuo, Avram J Holmes, Simon B Eickhoff, and BT Thomas Yeo. Local-global parcellation of the human cerebral cortex from intrinsic functional connectivity mri. Cerebral cortex, 28(9):3095-3 3114, 2018.
Oliver Schoppe, Nicol S. Harper, Ben D. B. Willmore. Andrew J. King, and Jan W. H. Schnupp. Measuring the performance of neural models. Frontiers in Computational Neuroscience, Volume 10 2016, 2016. ISSN 1662-5188. doi: URL htttps://wwx. Front Prontiersin. wtt.eeersiiiiooooonaaaaaaaaaaaaa 3389///o00.388//oo0 2016 00010
10. 10.3889ffncom.2016.001000 compotaationnl--eeerrseeeeeeeeee
Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib Majaj, Rishi Rajalingham, Elias B Issa, Kohitij Kar, Pouya Bashivan, Jonathan Prescott-Rov Franziska Geiger, et al. Brain-score: Which artificial neural network for object recognition is most brain-like? BioRxiv, pp. 407007, 2018.
Paul S Scotti, Mihir Tripathy, Cesar Kadir Torrico Villanueva, Reese Kneeland, Tong Chen, Ashutosh Narang, Charan Santhirasegaran, Jonathan Xu, Thomas Naselaris, Kenneth A Norman, al. Mindeye2: Shared-subject models enable fmri-to-image with hour of data. arXiv preprint arXiv:2403.I /207 2024.
Michael N Shadlen and William Newsome. Neural basis of a perceptual decision the parietal cortex (area lip) of the rhesus monkey. Journal of neurophysiology, 16(4):1916-1936, 2001.
Siddharth Srivastava and Gaurav Sharma. Omnivec2 novel transformer based network for large scale multimodal and multitask learning. In Proceedings of the IEEF/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 27412-27424, June 2024.
Marie St-Laurent, Basile Pinsard, Oliver Contier, Katja Seeliger, Valentina Borghesani, Julie Boyle, Pierre Bellec, and Martin Hebart. meeroood-thingee a large-scale fmri dataset for task-and data-driven assessment of object representation and visual memory recognition in the human brain. Journal of Vision, 23(9):5424 5424, 2023.
Barry Stein and Terrence R Stanford. Multisensory integration: current issues from the perspective of the single neuron. Nature reviews neuroccience, 94):255-266, 2008
13

---

Published as a conference paper at ICLR 2026
Jerry Tang, Meng Du, Vy Vo, Vasudev Lal, and Alexander Huth. Brain encoding models based on multimodal transformers can transfer across language and vision. Advances in neural information processing systems, 36:29654-29660 2023.
Mariya Toneva and Leila Wehbe. Interpreting and improving natural-lnguage processing (in machines) with natural language-processing (in the brain). Advances in neural information processing systems, 32, 2019.
Zhan Tong, Yibing Song, Jue Wang, and Limin Wang. Videomae: Masked autoencoders are data- efficient learners for self-supervised video pre-training. Advances in neural information processing systems, 35:10078-10092 2022.
David C Van Essen and Kamil Ugurbil. The future of the human connectome. Neuroinage, 62(2): 2012.
Brian A Wandell and Jonathan Winawer. Imaging retinotopic maps in the human brain. Vision
research, 51(7):718-75 2011.
Aria Y Wang. Kendrick Kay, Thomas Naselaris, Michael Tarr, and Leila Wehbe. Incorporating natural language into vision models improves prediction and understanding of higher visual cortex. BioRxiv, pp. 2022-09, 2022.
Weihao Xia, Raoul de Charette, Cengiz Oztireli, and Jing-Hao Xue. Umbrae: Unified multimodal brain decoding. In European Conference on Computer Vision, pp. 242-259. Springer, 2024.
Huzheng Yang, James Gee, and Jianbo Shi. arXiv:2308.00 01000.0175... 2023.
Memory encoding model.
arXiv preprint
Qiongyi Zhou, Changde Du, Shengpei Wang, and Huiguang He. Clip-mused: Clip-guided multi- subject visual neural information semantic decoding. arXiv preprint arXiv:2402.08994, 2024.
14

---

Published as a conference paper at ICLR 2026
Hyperparameter
Base value
Other values used for ensembling
Number of epochs
15
Number of TRs per window
100
Window jitter
10s
Learning Rate
10-4
Batch Size
16
Optimizer
A
Scheduler type
OneCycleLR (cosine)
Scheduler warmup phase
10%
Stochastic weight average epochs
8
Dropout
0
Weight Decay
0
Hidden Size
3072
Text model
Llama-3.2-3B
Audio model
Seav2Vec-Bert222
Video model
V-JEPA-2-Gigante-255
Loss
MSE
Pearson, SmoothI.1, HuberLoss 0.0, 0.4
Modality Dropout
0.2
Layer groups
[0.5, 0.75, 1] group by intervals concatenate concatenate
[0,0.5,1], [0.5, 11, [0. 0.2, 0.4, 0.6, 0.8, 1.] extract single layers average
Layer extraction
Layer aggregation
Modality aggregation
average X
Use subject embedding
Table 3: Hyperparameters used in our model
Text Audio
0.24
Video
0.22
0.20
0.16
0.14
0.12
0.0
0.2
0.4
0.6
0.8
1.0
Relative depth
Figure 7: Layerwise analysis of encoding scores. For each modality, we train unimodal encoders on top of the intermediate representations of the corresponding foundational model various relative depths (0 standing for the first layer and standing for the last layer of the model).
15

---

Published as a conference paper at ICLR 2026
0.10
Pita -earty
0.00
-0.10
Text
0.10
Plate Peady
0.00
00.05
-0.10
(b) Audio
0.20
Plate - Pearty
0.00
-0.20
Video
Figure 8: Late layers better encode high-level cortices than early layers. For each modality, we report the difference between the encoding score of the last layer and the first layer of the corresponding foundational model.
Attention paid to stimulus at time
0.07
0.06
0.05
0.04
0.03
0.02
0.01
-20
-15
-10
Time relative to current brain response (s)
Figure 9: Attention maps are reminiscent of the hemodynamical response function. For various times t spanning the stimulus window, we report the attention weight on the surrounding timesteps. The results are averaged over all in the window, and across all windows of the validation set.
16

---

Published as a conference paper at ICLR 2026
Encoding accuracy ODD, sub-01, movie-chaplin, mean accuracy: 0.2249
46-05
Encoadig accuracy COD, sub-01. movie-average, mean accuracy: 0.2381
0.66
0.33
05
Figure 10: Silent movies are better predicted in the visual cortex and less well predicted in the auditory cortex. Top: results for the first subject the Charlie Chaplin movie. Bottom: averaged across all OOD movies. Since the silent Chaplin movie belongs to the private test set of the competition, do not have access the ground truth labels and report screenshots of results provided by the competition host for the first subject.
17

---

Published as a conference paper at ICLR 2026
LICENSES
HuggingFace models:
Video-JEPA Apache
http::///gttttocccc////ceoeeeret blob/main/LIIENC
Wav2Vec-Bent-2.0:
MIT
http:://aggiinn23cc.c///ttt111
LLLama33223B: 1Lama-32-38:llamaa222 111.::l1mmmm12mmmm..........11 /
B-1B/bll0/maat/LLicES.....
Packages:
x transformers:
https://gtthbbb.ommmuucdaaaaaa X +
MIT
milearn: LICENSE)
BSD http:://hhhbicca///aaaaaaaaaaaah
Datasets:
Courtois NeuroMod: CCO httt:///reaattrrrrmoooooo/oo/ooa
hero/1.0/10 (/tr//L.0//egalcccee)
18


Reviews:
[{"review_id": "RLFRzXO0Qp", "reviewer": "Reviewer_DyxP", "summary": "The paper presents TRIBE, a model that predicts brain activity while people watch movies, using features from large pretrained models for text, audio, and video. By combining these three types of information and processing them with a transformer, the model can capture how different parts of the brain respond to complex, real-world stimuli. TRIBE achieves first place in a public competition, showing strong results across the whole brain, especially in regions that integrate multiple senses. The work suggests that using multimodal data and deep learning can help build more general models of brain function.", "questions": "See weakness.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Combines text, audio, and video features for brain prediction, which covers more brain regions than single-modality models.\n2. Achieves clear state-of-the-art results on a public benchmark, outperforming all other competitors by a noticeable margin.\n3. The model design is practical and scalable, using existing large models and a transformer to handle real-world, naturalistic data.", "weaknesses": "1. The model relies on combining a large number of models (ensemble), so it’s unclear how well a single model works in practice.\n2. Some details in the paper are inconsistent, for example the feature dimensions in Figure 2 don’t match the numbers in the methods section, and the number of teams is sometimes 262 and sometimes 263.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "0rHpZFhlkE", "reviewer": "Reviewer_dPCk", "summary": "This paper introduces TRIBE, a deep learning model that non-linearly integrates text, audio, and video features using a transformer to predict whole-brain fMRI responses to naturalistic videos. Its primary strength is its state-of-the-art performance, demonstrated by winning the Algonauts 2025 competition. The work is significant as it simultaneously addresses some key limitations of previous encoding models.", "questions": "1. The article mentions in the limitations section that the number of participants is currently small. Another concern I have is whether the model can be generalized to new subjects as discussed in [1]. If the brain decoding model is used in practice, it is unrealistic for each user to collect a large amount of their own data. With more modalities and increased data volume, can the model demonstrate its generalization ability across different subjects? (For example, a model trained on subjects 1-3 should also work for subject 4.)\n\n2. I also have doubts about the accuracy because this task involves predicting brain activity in response to different stimuli. However, even when the same subjects receive identical stimuli, their brain activity often varies, especially with temporal stimuli like audio and video. Since I lack the relevant background knowledge in neuroscience, I'm unsure if this mapping is reasonable. Nevertheless, this issue is more related to the competition's design. The authors can provide an explanation based on the specific circumstances.\n\n[1] Toward Generalizing Visual Brain Decoding to Unseen Subjects", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The motivation of this article is very good, as it analyzes the entire brain information from a multimodal perspective. It is of great practical significance and better aligns with the data processing procedures in the era of large models. Therefore, it is also very beneficial for researching more general brain foundation data.\n\n2. The experiments are well-conducted, and the performance is also good. Its 1st-place ranking out of 263 teams in a competitive benchmark is the strongest and most objective evidence for its effectiveness.\n\n3. Besides the model, this paper provides valuable neuroscientific insights, showing that multimodality provides the largest gains in associative cortices and reveals expected unimodal/bimodal dominance in different brain regions.", "weaknesses": "1. The article does not disclose or discuss the complexity of the model. To the best of my knowledge, many previous brain decoding projects employed relatively small models. However, the current method employs multiple pre-trained large models, and it should provide the overall size of the model so that others can evaluate and use it.\n\n2. Starting from line 48, the first two motivations actually involve many models that no longer use simple regression. Moreover, numerous recent studies have focused on multi-subject scenarios (e.g.,[1][2][3][4]), and it is essential to accurately describe the current research status .\n\n(In fact, the main weakness has been honestly stated by the authors in the \"limitations\" section.)\n\n[1] Mindeye2: Shared-subject models enable fmri-to-image with 1 hour of data\n\n[2] Umbrae: Unified multimodal brain decoding\n\n[3] CLIP-MUSED: CLIP-Guided Multi-Subject Visual Neural Information Semantic Decoding\n\n[4] Toward Generalizing Visual Brain Decoding to Unseen Subjects", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "oXEBS16pYN", "reviewer": "Reviewer_wQpp", "summary": "TRIBE presents and develops a neural network to predict whole-brain fMRI responses to naturalistic videos through integrated text, audio, and visual processing. The work addresses three critical limitations of prior encoding models: their reliance on linear mappings between AI and brain representations, subject-specific training that cannot leverage cross-individual similarities, and restriction to single modalities. The architecture extracts features from pretrained foundation models (Llama-3.2-3B for text, Wav2Vec-BERT-2.0 for audio, V-JEPA-2 for video), synchronizes them at 2 Hz, compresses layerwise representations, projects to a shared 1024-dimensional space, and processes concatenated multimodal embeddings through an 8-layer transformer with learnable subject and positional embeddings before mapping to 1000 cortical parcels.\n\nTraining on over 80 hours of fMRI per subject from the Courtois NeuroMod dataset, TRIBE achieves first place among 263 teams in the Algonauts 2025 competition, achieving a substantial margin over competitors. All 1000 parcels show significant prediction accuracy, and the model captures 54% of explainable variance on average, exceeding 80% in auditory and language cortices. Performance remains robust on highly out-of-distribution stimuli including animations, nature documentaries, and silent films. Ablations demonstrate that multimodality provides greatest benefits in associative cortices (up to 30% improvement over best unimodal baseline) while vision-only features slightly outperform in primary visual cortex. Both the transformer architecture and multi-subject training prove essential, and scaling analyses show continued improvement with more data and longer linguistic context without saturation.\n\nthe spatial averaging required for computational tractability likely discards retinotopic information critical for low-level visual encoding. Nevertheless, the work represents significant methodological progress by demonstrating that nonlinear integration of foundation model representations within a unified subject-general architecture can effectively predict whole-brain responses to complex naturalistic stimuli, providing a foundation for more comprehensive models of human cognition.", "questions": "1) Section 2.3 describes layer normalization after projecting each modality to 1024-D. For replicability, please clarify: (1) Are raw embeddings from each foundation model normalized (e.g., z-scored per modality over time) before the linear projection, or do they enter the projection layer unnormalized? Given different embedding dimensions and scales across models, this choice affects how the network learns to balance modalities. \n\n2) The paper tests multiple layer-grouping strategies (Table 3) but does not report which configurations perform best or where. Even a summary-level analysis would be valuable: Do models extracting early vs. late layers from each backbone receive differential ensemble weights in sensory vs. associative regions? I recognize that the 1000-model ensemble with confounded hyperparameters complicates attribution, but even coarse patterns (e.g., \"models grouping early V-JEPA layers achieved higher weights in V1\") would provide interpretability consistent with standard brain encoding practice. If full layer-wise decomposition is infeasible, reporting which layer-grouping strategies from Table 3 achieved highest overall performance can help inform architectural choices for future works.\n\n3) Table 2 shows substantial performance variation across stimulus types (0.32 for Friends vs. 0.17 for Chaplin). While Figure 4a provides modality ablations averaged across validation data, exploring whether multimodal benefits vary by stimulus characteristics could offer additional insight, e.g., whether audio contributes less for silent films. I recognize this may require re-running unimodal models on each test movie and that the limited sample (n=1 per genre) constrains statistical conclusions, but even illustrative case comparisons could be informative.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The model’s architecture is not a brute-force fusion but a theoretically motivated hierarchy: frozen modality experts (V-JEPA2, Wav2Vec2-BERT, LLaMA3.2) adapted into a shared latent space, followed by a transformer capturing temporal and intersubject alignment. The multisubject encoder and modality dropout mechanisms seem particularly well thought out, improving both biological plausibility and statistical efficiency\n\n2) The multimodal integration analysis is both methodologically clear and scientifically meaningful: ablations show monotonic gains from unimodal to bimodal to trimodal configurations (0.22 → 0.31 Pearson), empirically supporting the hypothesis that integrated sensory embeddings yield richer cortical alignment.\n\n3) TRIBE achieves a normalized Pearson correlation of ~0.54 ± 0.1 across 1,000 parcels, explaining more than half the variance in fMRI responses. The consistency across subjects and cortical regions is noteworthy,", "weaknesses": "1) V-JEPA activations are averaged over patches to manage compute, and the authors themselves expect degradation in low-level, retinotopic cortex. That choice complicates interpreting modality-dominance maps in early vision. It would make the paper stronger if the authors can provide some clarity here.\n\n2) The normalized Pearson metric (Eq. 1, claiming TRIBE captures \"54% of explainable variance\") relies on test-retest reliability (ρself) computed from only two repeated movies: Hidden Figures and Life. While I acknowledge that this follows standard practice in the field, Table 2 reveals substantial performance heterogeneity across genres: raw Pearson correlations span nearly 2× from Friends (0.32) to Charlie Chaplin (0.17). Given that different stimulus classes likely have different signal-to-noise characteristics (e.g., silent films vs. dialogue-heavy sitcoms, etc.), the reliability ceiling may also vary systematically across genres. Without demonstrating that Hidden Figures and Life yield consistent ρself estimates, or that their ceiling generalizes to the broader stimulus distribution, the \"54%\" figure may not accurately represent explainable variance for all content types in the test set. This perhaps limits the interpretation of normalized scores, particularly for out-of-distribution evaluation. A simple robustness check comparing ρself between the two repeated movies, or per-parcel ceiling variability, would  help to strengthen confidence in this metric.\n\n3)  The pipeline downsamples all modalities to 2 Hz, processes via transformer, and pools to TR (1.49s), but nowhere mentions hemodynamic response function (HRF) convolution. The HRF introduces ~5-6 second lag and temporal smoothing between neural activity and BOLD. While the transformer's large receptive field may implicitly learn such dynamics, three concerns arise: (1) downsampling audio from 50 Hz → 2 Hz may discard rapid transients (phonemes, prosody) that could improve prediction when properly lagged/smoothed to TR resolution, (2) interpretability suffers i.e. is the model learning neural dynamics requiring HRF or fitting BOLD directly?, and (3) no ablation (Table 3) tests whether explicit HRF convolution would improve performance. Given that competitors likely vary in temporal modeling choices, this could represent either a hidden performance limiter or an architectural shortcut that the transformer adequately compensates for but the paper provides no evidence to distinguish these possibilities.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)