The Human Brain as a Dynamic Mixture of Expert Models in Video Understanding
Authors: Christina Sartzetaki
Anne W. Zonneveld
Pablo Oyarzo
Alessandro Thomas Gifford
Radoslaw Martin Cichy
Pascal Mettes
Iris Groen
Venue: ICLR 2026 Poster (2026)
Keywords: representational alignment
Representational Similarity Analysis
RSA
benchmarking
neuro-AI
video AI
neuroscience
EEG
Primary area: applications to neuroscience & cognitive science

Abstract: The human brain is the most efficient and versatile system for processing dynamic visual input. By comparing representations from deep video models to brain activity, we can gain insights into mechanistic solutions for effective video processing, important to better understand the brain and to build better models. Current works in model-brain alignment primarily focus on fMRI measurements, leaving open questions about fine-grained dynamic processing. Here, we introduce the first large-scale model benchmarking on alignment to dynamic electroencephalography (EEG) recordings of short natural videos. We analyze 100+ models across the axes of temporal integration, classification task, architecture, and pretraining, using our proposed Cross-Temporal Representational Similarity Analysis (CT-RSA) which matches the best time-unfolded model features to dynamically evolving brain responses, distilling $10^7$ alignment scores. Our findings reveal novel insights on how continuous visual input is integrated in the brain, beyond the standard temporal processing hierarchy from low to high-level representations. After initial alignment to hierarchical static object processing, responses in posterior electrodes best align to mid-level temporally-integrative action features, showing high temporal correspondence to feature timings. In contrast, responses in frontal electrodes best align with high-level static action representations and show no temporal correspondence to the video. Additionally, temporally-integrating state-space models show superior alignment to intermediate posterior activity, in which self-supervised pretraining is also beneficial. We draw a metaphor to a dynamic mixture of expert models for the changing neural preference in tasks and temporal integration reflected in the alignment to different model types across time. We posit that a single best-aligned model would need such training and architecture as to allow combining and dynamically switching between these capacities.

Published as a conference paper at ICLR 2026
THE HUMAN BRAIN AS A DYNAMIC MIXTURE OF EXPERT MODELS IN VIDEO UNDERSTANDING
Christina Sartzetaki Anne W. Zonneveld" Pablo Oyarzo², Alessandro T. Gifford Radoslaw M. Cichy?, Pascal Mettes', Iris I.A. Groen'
Innorrattics Institute, University of Amsterdam, The Netherlands Department of Education and Psychology, Freie Universität Berlin, Germany {c. sartzttaaa,, a.w. wwwzoneeeed, i i.i.a. groon. groeeeeeuaa.... @uva.nl
ABSTRACT
The human brain is the most efficient and versatile system for processing dynamic visual input. By comparing representations from deep video models to brain activ- ity, we can gain insights into mechanistic solutions for effective video processing. important to better understand the brain and to build better models. Current works in model-brain alignment primarily focus on fMRI measurements, leaving open questions about fine-graines dynamic processing. Here, we introduce the first large-scale model benchmarking on alignment to dynamic electroencephalogra phy (EEG) recordings of short natural videos. We analyze 100+ models across the axes of temporal integration, classification task, architecture, and pretraining, us- ing our proposed Pemso-eempoal Representational Similarity Analysis (CT-RSA) which matches the best time-unfolded model features to dynamically evolving brain responses, distilling 10" alignment scores. Our findings reveal novel in- sights on how continuous visual input is integrated in the brain, beyond the stan- dard temporal processing hierarchy from low high-level representations. After initial alignment to hierarchical static object processing, responses in posterior electrodes best align mid-level temporally-inteeratve action features, showing high temporal correspondence to feature timings. In contrast, responses in frontal electrodes best align with high-level static action representations and show no temporal correspondence to the video. Additionally, cemporally-integratin state- space models show superior alignment to intermediate posterior activity, in which self-supervised pretraining is also beneficial. We draw a metaphor to a dynamic mixture of expert models for the changing neural preference in tasks and temporal integration reflected in the alignment to different model types across time. We posit that a single best-aligned model would need such training and architecture to allow combining and dynamically switching between these capacities.
INTRODUCTION
Humans are able to perceive and understand a highly dynamic world efficiently, with neural repre- sentations changing dynamically in response to continuous visual information. The framework of representational alignment (Sucholutsky et al. (2025) provides rich resource to investigate how hu- mans achieve this and for guiding model design. Within computational cognitive neuroscience, this framework is used to identify the mechanisms giving rise to cognition through hypothesis testing with task teek-pefforming computational models (Kriegeskorte with deep neural network computer vision models being the best models capturing neural responses in visual cortex (Yamins et al. 2014 Güiçüü & Van Gerven 2015). Conversely, machine learning can draw from cognitive neuroscience to inform more efficient and robust human-likk like artificial intel- ligence, e.g. through implementations of brain-like energy constraints (Lu et al.) (025a]), feedback pathways (Konkle & Alvarez (023), human-like development (Lu al. 2025b), perceptual straight- ening (Hénaff et al.] 20199] |Bagad & Zisserman, 2025), or even explicit finetuning on neural or cog- nitive data (Muttenthaler et al. 2025 (Safarani et al. 202T Moussa et al. 2025). Computational cognitive neuroscience has employed large-scale benchmarking as a tool for systematic and repro- ducible comparisons of model-brain representational alignment (Conwell al. 2024) under natural
Douglas 20188] Doerig et al.. 2023).
'These authors contributed equally this work.

---

Published as a conference paper at ICLR 2026
model unfolddd representations
Cross-Temporal Representational Similarty Analysis (CT-RSA) between models and brain
eig Kosption, vit Swin
Static Models
eg MileoWEE MVV..ED Stowfast Temporally-integratin Models
Socarmen's
allindde lime poins and aedall trmeggents
Lards
Porte
leot
Humaa Brain
Figure Our method for evaluating alignment 100+ models with dynamic EEG responses to video. The extraction of time-unfolded representations from both systems (left) is followed by CT- RSA (right), which computes the maximum model-brain alignment across all timepoints and layers of model. We systematically vary the axes of temporal integration, classification task, architecture, and pretraining here we highlight differences in model temporal integration.
mostly static image- stimuli, following the introduction of neural benchmark datasets (Schrimpf
al. 2018 Cichy et al., 2019: 2021 Gifford al., 2023 2024 Allen et al. 2022 Hebart al.
2023
Lahner et al.) 2024). However, static imgess lack temporal context, which strongly affects
visual processing (Willems & Peelen 2021), as for example illustrated by the temporal adaptation
of neural responses (Benda 2021 Brands al. 2024). Critically, findings from static image percep- tion do not automatically generalize to real-world conditions (Russ af., 2023 David al. 2004).
This highlights need for large-scale benchmark initiatives for dynamic video stimuli.
Recent work benchmarked model alignment to video fMRI (Sartzetaki et al. 2025 Garcia et al, 2025 Tang et al., 2025), showing how key axes of variation in those models (e.g. architecture, task training, temporal integration) affect brain alignment. However, these works did not yet consider two crucial factors: the rich dynamics that characterize neural responses, accessible non-invasively in humans via M/EEG, as well as the dynamics of temporally unfolded model features. this study, employ large-scale alignment benchmarking of deep neural network models dynamic human brain responses to natural videos using EEG, and leverage unfolded model features through our proposed Cross-Temporal Representational Similarity Analysis (CT-RSA). Our contributions:
We present the first large-scale representational alignment benchmarking on dynamic brain re- sponses (EEG) to natural videos, evaluating 100+ static and cemporally-interratte deep neural network models using matches the best time-unfolded model features across the EEG time-course.
Representational Similarity Analysis (CT-RSA), which
Our results reveal a changing neural preference for semantic tasks and temporal integration over time, and preferential specialization across the brain hierarchy tivity evolves from alignment with static, hierarchically increasing object processing to mid-level action representations, continuously tracking the video through temporal integration whereas frontal activity is best captured by early, static semantic action representations.
custerior-frontal Posterior ac-
Leveraging the time-unfolded model features, we show strong temporal correspondence be- tween model time and EEG time in posterior activity, but not in frontal activity.
Comparing the additional axes variation of model architecture and pretraining within mod- els of the same task and temporal integration, we show that state-space models better align to intermediate posterior processing, which self-supervised pretraining also beneficial.
Overall, our results suggest that neural representations reflected in fine- mee-grained dynamic brain mea- surements not best captured by any single DNN model type but rather resemble dynamic mixture of expert models that allows switching between semantic tasks and temporal integration, revealing opportunities for brain-inspired representation learning videos.
2

---

Published as a conference paper at ICLR 2026
2 RELATED WORK
Dynamic brain measurements under static and dynamic natural stimuli. Human brain dynam- ics are studied different timescales with distinct methods. For short timescales, usage of M/EEG is essential to make millisecond-lewe inferences, whereas the sluggish hemodynamic response under- lying IMRI prevents such precision. For static stimuli (i.e., images), findings from M/EEG studies of brain dynamics show that discriminable representations emerge as quick as within 50 ms after stimulus onset (Cichy et al. 2014). due a fast feedforward sweep of visual information processing from early to later stage brain areas, i.e. the cortical hierarchy (Serre et al. (207)) Early neural pro- cessing of static images has been found reflect low-level visual features (Groen et al. 2013), while later neural processing to reflect high-level representations, e.g. categorical processing (Cichy et al.) 2014) and task performance (VanRullen & Thorpee 200T), illustrating gradual emergence high- "eerarchy"(Bankonn (Bankson al., 2018; Cichy et al. 2016). This has not only been observed for static objects, but also for static scenes (Greene & Hansen 2020) and static actions (Zimmermann et al.] 2025). From these studies, high-level action informa- tion appears to emerge later (250 ms) than high-level object and scene information (150 ms). This may reflect need for integration of contextual information for action recognition (El-Sourani al., 2018). with mid-level features like motion patterns, objects, and scene information being processed in parallel before being integrated at the level of action semantics (Zimeerman et al. 2025)
level representations over time: "temporal hierarchy"
Human brain dynamics in response to short dynamic stimuli (i.e., videos) are comparatively much less explored. Using fMRI [ahner [ahner al. (2024) found hierarchical correspondence between action recognition model layers and brain regions as in static images, also demonstrating different levels of sensitivity to frame-shuffling and neural temporal correspondence
the video especially in early visual cortex (EVC)- consistent with prior findings on the hierarchy of temporal receptive windows
(Hasson et al.) 2008 Lerner et al. 2011). Additionally, work by Jung et al. (2025) showed that high-level semantic action features uniquely explain activity over a widespread range of cortex dur- ing dynamic natural vision, more so than semantic agent or scene features. A few works have used EEG to investigate brain dynamics to natural videos using domain-speciff social interaction videos; Dima et al. (2022) found a temporal hierarchy of visual, action-related and social-affective features, while McMahon et al. (2025) showed that mid and high-level social features are decodable at simi- lar timings, using EEG-IMR fusion. Using engine-rendered (naturalistic) short videos, |Krapetian al. (2255) also found temporal processing hierarchy and revealed that motion and action-related features are processed faster during video than static frame presentation. However, large-scale nat- ural video EEG datasets have been missing so far, limiting research on the fine-grained temporal dynamics of real-world visual processing, In this work, we are diving in underexplored territory by leveraging newly collected, large-scale high temporal resolution EEG dataset for nomain-genern (i.e. not -ecia--spccffic and natural (i.e. not rendered) dynamic stimuli.
Benchmarking model representational alignment on short-video fMRL. Very recent works have focused on large-scale benchmarking of model-brain alignment for videos. Garcia et al.] (2025) used fMRI of short social interaction videos, comparing 8 video action recognition models with 348 image object recognition models. They found that image models captured representations in EVC and the ventral stream, whereas video models outperformed in the lateral stream, linked to social cognition. Sartzetaki et al. 2025), used fMRI of short domain-genera videos, comparing 47 video models, 41 image object recognition models and 11 image action recognition models to disentangle temporal integration from classification task effects. Temporally-integnaai models in EVC, through better alignment of middle model layers, while action recognition models surpassed object ones in later stage brain regions via classification layers. Tang et al. (2025), used 5 different video fMRI datasets and compared 92 models, including image object recognition models, video action recognition models, multimodal models, and non-NN baselines. Through correlating brain alignment with model zero-shot performance in different tasks, they found that appearance- free (motion-only) action recognition and object recognition are the two most relevant tasks for brain alignment, with task-agnostic self-supervised models performing best at both tasks and alignment. Current studies are however all based on IMRI, precluding conclusions about fine-grained temporal dynamics. Similarly on the model side, the temporally-evolvin internal feature dimension remains unexploited. In this work, we present the first large-scale investigation of model alignment with highly dynamic brain representations recorded via EEG during short video viewing. Building on the model set and axes of variation from /Satzetaki et al.] (2025), we compare how different model groups and their internal temporal representations align with the brain across processing time.
models surpassed static
3

---

Published as a conference paper at ICLR 2026
3 METHODOLOGY
Figure I shows an overview of our methodology for measuring alignment of models to dynamic brain response We base our methodology on Sartzetta et al. (2025) but extend it in: (1) applying it to an EEG instead of fMRI dataset, (2) expanding the model architectures sampled, (3) unfolding the temporal dimension of the extracted model features and (4) proposing temporal extension to the representational alignment metric. We describe these aspects in the next three sections.
3.1 CROSS- SROSS-TEEIE TESSORAAML REPRESENTATIONAL SIMILARITY ANALYSIS (CT-RSA)
Neural network models compute representations at subsampled frame intervals, while the mapping of specific frames onto brain responses is uncertain, raising the question of how to align model and brain representations over time. We extend Representational Similarity Analysis (RSA) (Kriegesko- rte et al. 2008) to compare all temporally unfolded model features with all EEG timepoints without imposing assumptions about the relationship between the two, performing a cross-temporal match- ing that identifies the maximally aligning model timepoint at each EEG timepoint. Such cross- temporal matching also allows studying dynamic prediction (de Vries & Wurm 2023) and temporal generalization (King & Dehaene, 2014). We choose to build on RSA to perform a multivariate analysis that benchmarks emergent brain alignment of model representations.
Temporal Representational Dissimilarity Matrices. (A) EEG. At each EEG timepoint fy create a super-subject brain Representational Dissimilarity Matrix (RDM) (Beg), ) using a particular electrode subset with repetition- channel vectors Un for each participant (subject) We compute the super-subject RDM by averaging the RDMS of all subjects at that timepoint, as Brx- =
avg. (B.www B 1 r(v tNp U = Vi,j(i < 5), 0 S < K,O < iN < N where K the total number of videos and is Pearson correlation. (B) Models. We flatten the features of each model layer I and timepoint (M into a one-dimensional feature vector of length CxHxW, and reduce its dimensionality using Sparse Random Projection followed by Principal Component Analysis to 100 Principal Components to obtain fi- We compute one RDM per model layer and model tomnt as M
r(fi- P ) Vi,j(i < j), 0 j K, 0 S tm < M.
Cross- Temporal correlation of RDMs. To calculate the cross-temporal representational align- ment score between the model and EEG timecourses of RDMs, we first compute Spearman's p for all combinations of model and EEG timepoints for each layer Ri-txtt p( Bry, M)-(x)) We then choose the highest-corelating model layer and model timepoint per EEG timepoint Rax = mazi-ta ( This maximization is essential to retrieve insights from approxi- mately 1k RSA scores that result from all combinations of model layers and timepoints. Over all EEG timepoints, electrode subsets, and models, this allows us to handle over 10" RSA scores.
Noise ceiling computation. Because of individual subject variability brain data, noise ceilings for each electrode subset are computed to compare model RSA scores against the maximum obtainable score given the inter-subject variability, following standard approaches the field (Nili et al. 4). For the lower noise ceiling (LNCC) we compute mean RDM across all subjects except one. Then we take the Spearman correlation of the left-out subject RDM and mean RDM, repeat for all the subjects and calculate the average. For the upper noise ceiling (UNC) we take the mean of all RDMs without removing subjects, compute the Spearman correlation of each subject RDM with the mean RDM, and average. The UNC signifies perfect correlation for the amount of noise in the data, often referred to as the maximum amount of variance that can be explained. All further reported alignment scores are scaled by the UNC, and so is the LNC.
Statistical significance. To test if model RDMs correlated significantly with the brain RDMs, we performed permutation tests (Nili al. 2014), similar fashion as done in Sartzetaki et al. (025)). For each model RDM at every EEG timepoint we select the model RDM for highest-correlatngg
layer and model timepoint pair and permute rows and columns 10000 times using the same 10000 random permutations. We then calculate a null distribution per EEG timepoint by computing the Spearman correlations of all permuted RDMs with the brain RDM. For significance of group of
Due licensing restrictions of the stimuli used in the brain dataset, the video frames shown in the figure are sourced from representative videos captured by the authors themselves and are not subject to copyright. We incorporate CT-RSA in the Net2Brain python library (Bersch et al. 2025), see recent release https: htap::://githuue com/ evai-roro g- 1ab/Net2krain
4

---

Published as a conference paper at ICLR 2026
models against zero, we perform a two-tailed sign test between the average null distribution of all models in the group and the average observed Spearman correlation, corrected by subtracting the average observed Spearman correlation before stimulus onset, to account for inflated pre-stimulus correlation scores arising from the maximization inherent to our CT-RSA method. To test for sig- nificant differences between two groups of models, we perform two-tailed sign test between the null distribution created from the across-group differences in the within-group average distributions, and the observed difference in the average of the two model groups' Spearman correlations. When showing group medians instead of means, the above statistical inferences are performed on the me- dians. All statistical inferences are corrected for multiple comparisons across time points using FDR correction (Benjamini & Hochberg 1995) with a cluster threshold of two consecutive time points.
3.2 VIDEO MODELS
We expand upon the original set of 99 video models as used in Sartzetaki et al. (2025) to addition- ally reflect changes in the current state-of-the-art such the introduction of state space models (SSMs) as a scalable linear-complexity alternative to Transformers (Gu & Dao 2024), for both static vision (Zhu et al.) (024) and video (Li et al. 2024). With the addition of 3 VisionMamba object recognition and 8 VideoMamba action recognition models, our final set of models includes 44 image models trained for object recognition on ImageNet, 10 image models trained for action recognition on Kinetics 400, and 49 video models trained for action recognition on Kinetics 400. Image action models are trained on videos but treat time in trivial way (separate computations per frame). An additional set of 7 models trained for action recognition on other datasets (Kinetics 710 and Something-SSmmething Something-v2) was also evaluated, bringing the total to 110 computer vision models tested overall. Comprehensive and extended lists of models can be found in Appendix B
Temporally unfolded feature extraction. Since the models process a fixed number of frames at a given sampling rate, each video is divided into S sub-clips of length T, differing per specific model. For each layer, we extract features of shape (T, C, H, W) from each sub-clip, and unfold the temporal dimension to obtain TxS=M model timepoint features of shape (C, H, W). We extract features from all higher-level blocks in the models and include the final classification layer.
3.3 EEG DATASET
We utilize the newly collected EEG Moments Dataset (EEGMD), an EEG-based extension of the large scale video fMRI BOLD Moments Dataset (BMD) (Lahner et al., 2024). EEGMD covers the exact same set of 1 1102 short (3s) natural videos from Moments in Time (Monfort et al. 2019 2021). See Appendix [A] for more information on EEG acquisition. The dataset consists of EEG recordings of 6 participants for a "train set" of 1000 videos with repeats per video and a "test set" of 102 videos with 24 repeats, using a set-up of 128 electrodes. Data was recorded at a sampling rate of 1000 Hz with online filtering (between 0.1 Hz and 100 Hz) and rereferenced (to Fz). In the current analysis we use the test set for the application of RSA, signal to noise ratio is higher due the greater number of stimulus repetitions to average across (see decoding results Appendix representative of the whole
C Fig. [)) similar to Sartzetaki et al. (2025). The test set videos dataset, covering a wide range of objects, actions and scenes (for more elaborate description of the video contents see [ahner et al. (2024)) All main analyses
based the group-average level; we provide additional supplementary results for subject level analyses in Appendix Fig. 9
Preprocessing and electrode selections We performed offline preprocessing using Python and MNE (Gramfort et al., 2013). The continuous EEG data was epoched into trials from -0.2s to 3.5s with respect to stimulus onset, baseline-correeted by subtracting the mean of the pre-stimulus period separately for each trial and channel, and temporally downsampled to 50 Hz. Next, we performed multivariate noise normalisation (MVNN) based on the covariance matrices of each timepoint to reweigh (un)reliable sensors and to (de)emphasise specific spatial frequencies, as recommended for multivariate analyses such as RSA (Guggenmos al. (018). Our analyses are based on two electrode partitions. First, we select posterior electrodes (35), which overlay visual cortex and are commonly used in vision studies (Xie et al.. 2020 Seijdel et al., 202 Loke et al. 2024). To make coarse distinction between the opposite ends of the brain's 's spatial organization, we select a second partition of frontal electrodes (54), covering (fre)frontal cortex associated with executive functions. The inclusion of frontal electrodes is an exploratory choice, further motivated by the lack of prior knowledge on the localization of visual processing during longer timescales (video). See Appendix for the names of all included electrodes.
5

---

Published as a conference paper at ICLR 2026
Image Models [ImageNet-I] 1k)
Image Models (Kinetics-400)
Video Models (Kinetics-400)
Posterior Electrodes
Frontal Electrodes
UNC scaled Sptarann' tho
III
IV
II
IV
0.4
Time (soconds)
Timo Isaconds)
Layer doath
Model une point
Video
video
Figure 2: (Left) Task and temporal integration types dynamically interchange in posterior electrodes. (Right) Action task specificity is early and stable in frontal electrodes. Row (a) shows the maximum score over all model timepoints and layers (group average), with significance marked by squares (against zero) and two-colored circles (pairwise), while (b) and (c) show the layer and timepoint that yield the scores. Processing stages 1-IV are identified and LNCs are outlined in gray background.
RESULTS
We assess brain alignment over time for 110 deep neural networks using CT-RSA, considering four main of variation: temporal integration, classification task, architecture, and pretraining.
Task and temporal integration types dynamically interchange in posterior electrodes. In Fig. (Left: Posterior) we compare models across the whole EEG time-course by varying (1) the temporal integration, i.c. image (static) v.s. video (temporally-ing -inporally-integratinng) models, and (2) the classification task, i.c. object (ImageNet-Ik) v.s. action recognition (Kinetics-400) models. RSA scores (Fig. ) in all model groups are significant against zero from 0.08s, extending to the duration of the whole video and offset. Based on visual inspection of the development of these scores, we distinguish four temporal stages of processing: (I) 0.06s 0.24s, (II) 0.24s 0.8s, (III) 0.8s 2s and (IV) 2s 3s. To summarize the results these different stages and explore model variation, we additionally evaluate bin-averaged scores in Fig. 3 We next discuss these stages in tum, simultaneously interpreting the score (Fig. 3)) and the best model layer (Fig. 2p(3p) that gives rise it. First in stage scores peak for all model groups at 0. and image models significantly outperform video models, with the best model being AlexNet. For all model groups this high correspondence can accounted for by relatively early model layers (below 0.5), and the peak in scores decays, by progressively later layers. This suggests that during this stage, posterior processing reflects static low-level information, independent of task. In stage II, object recognition models show clear peak, outperforming the other model groups; the best model here is a DenseNet. In contrast with stage I, the significant scores for all model groups are due late layers, primarily relating posterior processing this stage to static high-level object information. During stage III, we observe steady decrease in image model scores, while the score for video models increases and then remains relatively stable, before dropping towards the end of the stage. This leads to video models outranking the other groups (best model being MViT-v2), with scores being driven by mid-level layers. Thus, overall stage III can be summarized scores and layer depth across all model groups remain stable after reaching their lowest point, with video models less significantly outperforming image models than in stage III. Looking across stages in Fig. [2b/3,, action recognition model scores correspond to an earlier layer than object recognition ones in stages I and III, with less clear patterns in the other stages.
2
mid-level femporally-integraieee
action processing stage. During stage IV,

---

Published as a conference paper at ICLR 2026
Posterior Electrodes
Frontal Electrodes
UNC scaled Speamans mo
0.5
Image Models (ImageNet-1k)
Image Models (Kinetics-400)
Video Models (Kinetics-400)
AAachtt
Danselit
TSM Deniphet
TSM HobillOee
0.0
Layer depth
85
Model time point
0.5
11
IV
11
IV
Figure 3: Bin-averaged results showing model variability within each of the temporal processing
stages in Fig. [ outlining best-aligned models in row (a). Significance is marked by star at (against zero) and by the respective number of stars for 0.05, 0.01, and 0.001 on top (pairwise).
The best model timepoint (Fig. 3 reveals strong correspondence between model and EEG timepoints in posterior electrodes: early EEG timepoints best correspond to early model timepoints (close 0), and later EEG timepoints to gradually later model timepoints (yet only up ~0.6), highlighting the temporality of posterior neural responses. Note also that where layer depth is high (stage II), the match in model timepoint less strict.
The analysis demonstrates that posterior processing during video perception is highly temporal, progressing through distinct stages that emphasize different representational demands. An early peak around 0.14s reflects low to mid-level static processing, followed by rapid shift toward higher- level static object representations. Beyond this point, posterior parts of the brain gradually transition into mid-level lemporally-integratii action processing, marking the late stages of video perception.
Action task specificity is early and stable in frontal electrodes. In Fig. (Right: Frontal) we show results for the frontal partition. We observe that most neural processing of video information occurs in stages and II, as model groups show no significant alignment scores afterwards. Scores in this period show two peaks for all model groups, at 0.24s and 0.5s, with static action recognition models significantly outperforming the other groups at the last peak. This results from correspon- dence with late model layers. Regarding the best model timepoint (Fig. (2/3) and in contrast to posterior electrodes, we see that in frontal electrodes, during the period of significant scores, there clear temporal correspondence between model and EEG timepoints with large spread across models (Fig. BD). These results suggest that during video perception, neural processing reflected frontal electrodes shows limited temporality and primarily engaged until 0.8s with high-level, action-related information that is largely independent of within-video dynamics.
Context window and sampling rate influence alignment in late posterior processing. In Fig. we show the effects of model context window length (num. of frames) and sampling rate (fps) on brain alignment. We observe significant positive correlation between these factors and posterior brain alignment for temporally integrating models during stages III and IV, during which temporally integrating models also show the highest alignment (Fig. pb. These findings reinforce the notion that stage posterior processing of vidcos relies temporal integration, with larger temporal context windows and higher temporal resolution enhancing alignment.
State-space video models best capture mid-level intermediate posterior processing. In Fig. we compare brain- aliin-al different architectures, i.e. CNNs, Transformers, or SSMs, focusing on the stages I-IV identified in the previous sections. To avoid confounds from task temporal inte- gration, we focus this analysis on video action recognition models. SSMs are the most brain- aaiine posterior channels in stages I-II, especially in stage II, and through earlier layers (mid-depth).

---

Published as a conference paper at ICLR 2026
Posterior Electrodes
Frontal Electrodes
CNNs
Transforers
State Space Models
Layer depth
6.5
IV
III
IV
Figure 4: Architecture variation in bin-averaged scores for each stage. (Left) Video SSMs best capture mid-level intermediate posterior processing. (Right) CNNs give slight edge in alignment to high-level frontal processing.
Additionally in stage I, Transformers are significantly better aligned than CNNs. In frontal channels CNNs most brain-aligned in stages I-II via late layers, however, only with marginal difference to Transformers. In stages III and IV of both channel sets differences are mostly not significant. In App. C we show additional comparisons that a) control for model pretraining (Fig. 10 and
b) compare between object recognition instead of video action recognition models (Fig.
The
effects of architecture are stable across pretraining types, but differences between object recognition models are more muted. Specifically, static object SSMs do not show the observed advantage for video SSMs posterior stage II. This suggests that SSMs capture a distinet component of neural processing in posterior channels during phase II, previously linked to static high-level object rep- resentation, through comporally-interrann disadvantage of CNNs in early processing and an advantage of SSMs in intermediate posterior processing. Considering their architectural differences to Transformers, the CNN disadvantage sug- gests global attention is useful for early alignment, while the SSM advantage suggests recurrent processing benefits alignment to subsequent intermediate responses. This latter advantage appears more related recurrency across time than across patches, for static SSMs it does not persist.
mid-level action features. Overall, these results show
Self-supervised to no pretraining switch in posterior electrodes. In Fig- 5 we compare the brain alignment of video action recognition models having different types of pretraining, either no pretraining, supervised pretraining on images (object recognition), supervised pretraining on videos (action recognition), or self-supervised pretraining on videos. We observe that in stages I-II of the posterior electrodes self-supervised pretraining achieves superior brain alignment. In stage it is on par with pretraining on image object recognition, while in stage II is superior to all other types. In stage III, no pretraining is significantly better than all other types. In stage IV in posterior and in all stages in frontal electrodes, significant differences are less consistent, with only a slight advantage of supervised video pretraining in frontal. We again further control for the model architecture in App- Fig. [3 and find that the advantages of self-supervised pretraining in stage II and no pretraining stage III are robust across architectures. A conclusion could draw from this analysis, is that the advantage of self-supervised pretraining in the primarily "object processing" stage relates to self-supervision enabling generalization other tasks (through task-general pretraining objectives like spatiotemporal patch reconstruction), while the benefit of no pretraining in the temporally- integrative stage reflects avoiding the shortcut learning of unrelated patterns that is common during pretraining (Byvshev et al. 2022). We additionally compare action recognition fine- finettuning datasets (Sth-Sth-v2. K710), showing no effects on alignment at any stage or partition (App. Figs. 14 15
UNC-scaled SpearmanSshho
0.5
Posterior
No pretraining
Object--upervised images
Frontal
3.4
Action surtro-e on videos
Self- on videos
0.3
3.2
0.3
3.0
IV
Figure 5: Pretraining variation in bin-averaged scores for each stage. (Left) Switch of benefit from self-supervised (stage II) to pretraining (stage III) in posterior electrodes. (Right) Slight advan- tage of supervised video pretraining in frontal electrodes.
8

---

Published as a conference paper at ICLR 2026
5 DISCUSSION AND CONCLUSIONS
In this work we performed a large-scale model comparison of both static and deep neural networks to dynamic EEG recordings. Using CT-RSA make the following main observations: (1) Neural processing during video perception in posterior parts of the brain is highly temporal and unfolds in distinct stages, best captured by different type of representations: from an initial alignment with static low-level features, to mid- and high-level object features, and finally to mid-level temporally integrative action features. (2) In contrast, neural processing during video perception in frontal parts of the brain shows restricted temporality and is best aligned with static high-level action features early in the video. (3) Architecture-wiss
action SSMs best capture mid-level representations in posterior electrodes, reflecting a different component of the response in the primarily object-related stage, while indicating potential benefits of global attention and recurrent temporal processing for alignment to carly-intermediate responses. (4) Self- supervised pretraining helps capture brain representations in the early, primarily object-related stage, while performing pretraining captures brain representations of later more temporally- -inteegaatit stages, indicating independent benefits of general pretext tasks and unbiased target domain features.
samporally-integraln
What we learn by moving to dynamic natural stimuli. Moving from static images to more real world dynamic videos, we find similarities and differences in neural processing between these two formats. Congruent with the works of Bankson et al.] (2018), Greene & Hansen (2020) and ZZm- mermann et al. (2255) in the domain of image perception, we find temporal hierarchy of object and action features in posterior processing, but only within the duration of stages I and II, i.e. the first 500 ms of processing, consistent with the stimulus durations in those studies. Our findings are also largely consistent with prior work on the neural dynamics of short videos showing a temporal hierarchy of (social) action-related features (Dima et al.) 2022, McMahon et al. 2025 Karapetian et al. 2025). where low and high level object features emerge earlier than action features. However, as we extend beyond the timescales used in these studies (beyond 1s) we discover that mid-level temporally-integrativ action features are most brain-aligned until the end of the video, challenging the notion of a strict temporal hierarchy. The critical role of temporal integration during this late- stage processing is further highlighted by the observed increase in alignment with larger context windows and higher temporal resolution. Notably, this sustained alignment occurs consequently after the peak in alignment with regards to static high-level action features in frontal areas, raising the possibility that feedback from frontal to posterior regions may contribute to shaping later stages of visual processing of dynamic action information. Multiple studies have posed that during ob- ject recognition in static context feedback information from prefrontal cortex to posterior regions is critical in shaping behaviorally sufficient object representations, especially under challenging con- ditions (Goddard et al., 2016; Kar & DiCarlo 2021 Oyarzo et al., 2025). Specifically, Oyarzo et al. 2025) pose frontal cortex plays a corrective role under these conditions, reshaping high-level vi- we extrapolate, video processing can be viewed as a more demanding extension of static image processing. This increased complexity may explain the recruitment of frontal activity, which appears to contribute early on, followed by a reconfiguration of representations in posterior activity for updated mid-level feature processing.
sual representations resolve ambiguity.
What we learn from using dynamic brain measurements for benchmarking. Compared to prior work video fMRI aligmment benchmarking (Sartzetaki et al. 2025). which found high alignment of cemporally-integratii
video models through mid-layers in EVC, we find an advantage for these model features now temporally localized in stages III and IV, i.e. from 0.8s and onwards, in the posterior part of the brain, following initial alignment with high-level object model layers. Com- bining the insights from both studies, we hypothesize that mid-level dynamic processing could be performed by the EVC after the initial encoding of the video appearance. The high alignment of action recognition models reported in high-level visual cortex by |Sartzetaki et al. (2025) was also observed in the current study, but in the frontal electrode partition, likely reflecting an even higher- level cortical stage of processing, during stage II (0.24s s). Additionally, leveraging the temporal resolution of EEG, we find that object representations also contribute to video processing in the pos- terior part of the brain, being the most predictive during early processing but rather briefly, before the processing of dynamic features. A hypothesis for the absence of object contribution in fMRI alignment for video is that, relative to the whole video time-course, object features are processed only briefly, and this transient signal is lost in the aggregation of the fMRI response. Similarly, the enhanced alignment with static processing during stage (0.06s-0.024s) in posterior electrodes may be too brief to appear in the fMRI signal. We can also compare our results for the other factors

---

Published as a conference paper at ICLR 2026
of variation that study explored, namely architecture (without SSMs) and training dataset. For chitecture they showed CNNs and Transformers being overall equivalent in fMRI, with differences only in which layer was best aligned; in CNNs the best aligning layer was a later layer (==0.6) than in Transformers (zx0.2), primarily in EVC. Here we mostly find the same; CNNs and Transform- align overall similarly apart from posterior stage I which shows strong overall advantage for Transfomers, and in posterior stages II. III, IV (especially II), the best aligning layer is a later layer in CNNs than in Transformers. In our results, this trend also continues in frontal electrodes. For training dataset, the fMRI study showed disadvantage of ego-centric datasets in alignment to face- processing brain regions, whereas we find no differences in alignment to EEG (Appendix [ Fig. 14 most probably due to the lack of fine-grained spatial resolution.
Prospects for model building. Our findings suggest that during video perception the brain per- forms computations that are not best represented by any single model during the entire EEG time- course, but rather by alternating semantic tasks and temporal integration strategies, resembling a dynamic mixture of expert models. This type of neural processing is also corroborated by studies of non-visual, higher cognitive systems like reasoning and the multiple demand system, with Mix- ture of Experts (MoE) models matching established brain pathways (Cook et al. 2025 AlKhamissi jet al.] 2025). We propose (1) that a single model could be best aligned to the whole time-course if it was trained a sufficiently general objective (e.g. self-supervised masked modeling) so that it can develop experts for object and action recognition, and for temporally-integrn processing (2) that dynamically switching between those experts is a design choice with potential for human-like capabilities (e.g. efficiency). Regarding (I), preliminary results on a single model (VideoMamba, App. Fig. T show that pure self-supervision does yield the highest alignment in stages I and II (potentially due to the general objective, as in Tang et al. (2025)) but is surpassed by the fine-tuned version at later processing stages, suggesting a further need for supervised training- Concerning (2), this dynamic switching might prove limited in block-based (i.e. CNN/Transformer) video models, as they process videos in fixed temporal blocks; any dynamic switching could then only occur within these time spans. Architectures better suited to support this property are recurrent neural networks (c.g. SSMs) as they process input sequentially, continuously updating their hidden states. Dynamic switching could then emerge implicitly be enforced, potentially reducing com- putational costs via keeping overall network activation low, or using feedback connections to make use of prior experts. This exemplifies the potential of brain alignment for model building, inviting the machine learning and cognitive neuroscience communities to think again in a more unified way through the search of common core principles in both engineering and biological solutions.
vs. static
Application prospects of CT-RSA. The usage of CT-RSA is not limited to the current setting. In principle, the method generalizes to any setting that involves comparing two forms of temporal. or rather sequential, representations, On the model side, this could be multimodal audiovisual and language representations across time (e.g. Gifford et al.] (2024), either from separate unimodal DNNs or multimodal DNNs, supporting analyses on when exactly in time multimodally integrative neural processing occurs rather than unimodal processing. On the neural side, it could likewise extend to long-format IMRI time series or other sequential neural measurements. For example in the set up of Ascoli et al. (2025), encoding could be replaced with CT-RSA as an option for exploring the inherent temporal model-brai alignment instead of maximizing neural predictivity. Related forms of cross-temporal matching have been used to study dynamic prediction (de Vries & Wurm 2023) and temporal generalization (King & Dehaene 2014) of representations.
Limitations and future work. While our setup involved 100+ models, extending SSarzetaki et al. (2025), specific model types should be investigated in more depth, such as purely self-supervised models and models with different types of recurrent mechanisms (SSMs, LRUs, RNNs). Addition- ally, fine-grained differences between video models can be further explored by using, instead of coarse top-down grouping, a bottom-up grouping that focuses on variations in temporal feature gen- cralization. Another exciting direction is collecting neural data with intermediate duration videos (10-20s) that could include elements of visual surprise, camera changes, or scene cuts. Lastly, given the limited spatial resolution of EEG, our ability to draw conclusions about the precise spatiotem- poral dynamics is constrained. To more directly assess whether, when and how the prefrontal cortex could be involved in the neural processing of videos, a promising direction for future work would be to employ model based EEG-IMRI fusion (Hebart et al., 2018 Cichy & Oliva 2020 Cichy et al. 2014), combining the current EEGMD with BMD (Lahner et al. 2024) and leveraging the the complementary spatial and temporal resolution of these different brain measurements.
10

---

Published as a conference paper at ICLR 2026
ACKNOWLEDGMENTS
CS is supported by an ELLIS Amsterdam Unit grant to IIAG, AWZ acknowledges support from the UvA Data Science Centre, as part of the Human Aligned Video Al Lab.
REFERENCES
Badr AlKhamissi, C Nicolò De Sabbata, Greta Tuckute, Zeming Chen, Martin Schrimpf, and An- toine Bosselut. Mixture of cognitive reasoners: Modular reasoning with brain-like specialization. arXiv preprint arXiv:2506. / 13331, 2025.
Emily J Allen, Ghislain Stey Yihan Wu, Jesse L Breedlove, Jacob S Prince, Logan T Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, et al. A massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence. Nature Neuroscience, 25(1):115 116::::6:1111111 126, 2022.
Piyush Nitin Bagad and Andrew Zisserman. Chirality in action: Time-aware video representation learning by latent straightening. In The Thirty-ninth Annual Conference on Neural Information Processing Systems, 2025.
Brett B Bankson, Martin N Hebart, Iris IA Groen, and Chris Baker. The temporal evolution of conceptual object representations revealed through models of behavior, semantics and deep neural networks. Neurolmage, 178:172-182, 2018.
Jan Benda. Neural adaptation. Current Biology, 3 110-R11,, 2021.
Yoav Benjamini and Yosef Hochberg, Controlling the false discovery rate: a practical and powerful approach multiple testing. Journal of the Royal Statistical Society: Series (Methodological), 57(1):289-300 1995.
Domenic Bersch, Martina G Vilas, Sari Saba-Sadiya, Timothy Schaumlöffel, Kshitij Dwivedi, Christina Sartzetaki, Radoslaw M Cichy, and Gemma Roig. Net2brain: A toolbox to compare artificial vision models with human brain responses. Frontiers in Neuroinformatiic, 19:1515873, 2025.
Amber M Brands, Sasha Devore, Orrin Devinsky, Werner Doyle, Adeen Flinker, Daniel Friedman, Patricia Dugan, Jonathan Winawer, and Iris IA Groen. Temporal dynamics of short-term neural adaptation across human visual cortex. PLoS Computational Biology, 201))11, 2024.
Petr Byvshev, Pascal Mettes, and Yu Xiao. Are 3d convolutional networks inherently biased towards appearance? Computer Vision and Image Understanding, 220: 103437, 2022.
Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman. A short note on the kinetics-700 human action dataset. arXiv preprint 1rXiv:1907.06998) 2019.
Radoslaw M Cichy and Aude Oliva. Am/ceg-fmri fusion primer: resolving human brain responses in space and time. Neuron, 107(5):772-781, 2020.
Radoslaw Martin Cichy, Dimitrios Pantazis, and Aude Oliva. Resolving human object recognition in space and time. Nature Neuroscience, 173))455 2014.
Radoslaw Martin Cichy, Aditya Khosla, Dimitrios Pantazis, Antonio Torralba, and Aude Oliva. Comparison of deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondence. Scientific Reports, 6(1):27755, 2016.
Radoslaw Martin Cichy, Gemma Roig, Alex Andonian, Kshitij Dwivedi, Benjamin Lahner, Alex Lascelles, Yalda Mohsenzadeh, Kandan Ramakrishnan, and Aude Oliva. The algonauts project: A platform for communication between the sciences of biological and artificial intelligence. arXiv preprint arXiv: 05667, 2019.
Radoslaw Martin Cichy, Kshitij Dwivedi, Benjamin Lahner, Alex Lascelles, Polina lamshchinina, Monika Graumann, Alex Andonian, NAR Murty, K Kay, Gemma Roig, et al. The algonauts project 2021 challenge: How the human brain makes sense world motion. arXiv preprint arXiv:2 13714, 2021.
11

---

Published as a conference paper at ICLR 2026
Colin Conwell, Jacob S Prince, Kendrick N Kay, George A Alvarez, and Talia Konkle. A large-scale examination of inductive biases shaping high-level visual representation in brains and machines. Nature Communications, 15(1):9383, 2024.
Jack Cook, Danyal Akarca, Rui Ponte Costa, and Jascha Achterberg. Brain-like processing pathways form in models with heterogeneous experts. In The Thirty-ninth Anmual Conference Neural Information Processing Systems, 2025.
Stéphane d' d'Accoli, Jérémy Rapin, Yohann Benchetrit, Hubert Banville, and Jean-Rémi King. Tribe: Trimodal brain encoder for whole-brain fmri response prediction. arXiv preprint arXiv:2507.22221 2025.
Stephen David, William E Vinje, and Jack L Gallant. Natural stimulus statistics alter the receptive field structure of v1 neurons. Journal of Neuroscience, (111)::99117006, 2004.
Ingmar EJ de Vries and Moritz F Wurm. Predictive neural representations of naturalistic dynamic input. Nature Communicationss 14(1):3858, 2023.
Diana C Dima, Tyler M Tomita, Christopher Honey, and Leyla Isik. Social-affective features drive human representations of observed actions. Elife, 11:e75027, 2022.
Adrien Doerig, Rowan Sommers, Katja Seeliger, Blake Richards, Jenann Ismael, Grace W Lind- say, Konrad P Kording, Talia Konkle, Marcel AJ Van Gerven, Nikolaus Kriegeskorte, et al. The neuroconnectionis research programme. Nature Reviews Neuroscience, 24(7):431-450, 2023.
Nadiya El-Sourani, Moritz F Wurm, Ima Trempler, Gereon R Fink, and Ricarda Schubotz. Making sense of objects lying around: How contextual objects shape brain activity during action observa- tion. Neurolmage, 06:4499-37,, 2018.
Kathy Garcia, Emalie McMahon, Colin Conwell, Michael Bonner, and Leyla Isik. Modeling dy- namic social vision highlights gaps between deep learning and humans. In The Thirteenth Inter- national Conference on Learning Representations, 2025.
Alessandro T Gifford, Benjamin Lahner, Sari Saba-Sadiya, Martina G Vilas, Alex Lascelles, Aude Oliva, Kendrick Kay, Gemma Roig, and Radoslaw Cichy. The algonauts project 2023 chal- lenge: How the human brain makes sense of natural scenes. arXiv preprint urXiv:2301.03198 2023.
Alessandro T Gifford, Domenic Bersch, Marie St-Laurent, Basile Pinsard, Julie Boyle, Lune Bellec, Aude Oliva, Gemma Roig, and Radoslaw M Cichy, The algonauts project 2025 challenge: How the human brain makes sense of multimodal movies. arXiv preprint arXiv:2501.00504 2024.
Erin Goddard, Thomas A Carlson, Nadene Dermody, and Alexandra Woolgar. Representational dynamics of object recognition: Feedforward and feedback information flows. Neurolmage, 128: 385-397, 2016.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE International Conference on Computer Vision, pp. 5842-5850, 2017.
Alexandre Gramfort, Martin Luessi, Eric Larson, Denis A Engemann, Daniel Strohmeier, Christian Brodbeck, Roman Goj, Mainak Jas, Teon Brooks, Lauri Parkkonen, et al. Meg and eeg data analysis with mne-python. Frontiers in Neuroinformatics, 7:267, 2013.
Michelle R Greene and Bruce Hansen. Disentangling the independent contributions of visual and conceptual features to the spatiotemporal dynamics of scene categorization. Journal of Neuro- science, 40(27):5283-5299 2020.
Iris IA Groen, Sennay Ghebreab, Hielke Prins, Victor AF Lamme, and H Steven Scholte. From image statistics to scene gist: evoked neural activity reveals transition from low-level natural image structure scene category. Journal of Neuroscience, 33(48): 188): 188244,, 2013.
12

---

Published as a conference paper at ICLR 2026
Albert Gu and Tri Dao. Mamba: Linear- time sequence modeling with selective state spaces. In First Conference on Language Modeling. 2024.
Umut Güçli and Marcel AJ Van Gerven. Deep neural networks reveal gradient in the complexity of neural representations across the ventral stream. Journal of Neuroscience, 35(27):10005-00011 2015.
Matthias Guggenmos, Philipp Sterzer, and Radoslaw Martin Cichy. Multivariate pattern analysis for meg: A comparison of dissimilarity measures. Neurolmage, 173:434 447, 2018.
Uri Hasson, Eunice Yang, Ignacio Vallines, David J Heeger, and Nava Rubin. A hierarchy of tem- poral receptive windows in human cortex. Journal of Neuroscience, 28(10)253392555555 2008.
Martin N Hebart, Brett B Bankson, Assaf Harel, Chris Baker, and Radoslaw M Cichy. The repre- sentational dynamics of task and object processing in humans. Elife, 7:e32816, 2018.
Martin N Hebart, Oliver Contier, Lina Teichmann, Adam H Rockter, Charles Zheng, Alexis Kid- der, Anna Corriveau, Maryam Vaziri-Pashkann and Chris Baker. Things-data, multimodal collection of large-scale datasets for investigating object representations in human brain and be- havior, Elife, 12:c82580, 2023.
Olivier Hénaff, Robbe LTT Goris, and Eero Simoncelli. Perceptual straightening of natural videos. Nature Neuroscience, 22(6):984-991, 2019.
Heejung Jung, Xiaochun Han, Ma Feilong, Jane Han, Deepanshi Shokeen, Cara Van Uden, Isabella Hanson, Andrew Connolly, James Haxby, and Samuel A Nastase. Action features dominate cortical representation during natural vision. bioRxiv, pp. 2025-01, 2025.
Kohitij Kar and James J DiCarlo. Fast recurrent processing via ventrolateral prefrontal cortex is needed by the primate ventral stream for robust core visual object recognition. Neuron, 109(1): 164-176, 2021.
Agnessa Karapetian, Alexander Lenders, Vanshika Bawa, Martin Pflaum, Raphael Leuner, Gemma Roig, Kshitij Dwivedi, and Radoslaw M Cichy. Investigating the temporal dynamies and mod- elling of mid-level feature representations in humans. bioRxiv, pp. 2025 03, 2025.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint #xXiv:/705.06550 2017.
Jean-Rémi King and Stanislas Dehaene. Characterizing the dynamics of mental representations: the temporal generalization method. Trends in Cognitive Sciences, 18(4):203-210, 2014.
Talia Konkle and George Alvarez. Cognitive steering in deep neural networks via long-range mod- ulatory feedback connections. Advances in Neural Information Processing Systems, 36:21613- 21634, 2023.
Nikolaus Kriegeskorte and Pamela K Douglas. Cognitive computational neuroscience. Nature Neu- roscience, 21(9): 148-1160, 2018.
Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandettini. Representational similarity analysis- connecting the branches of systems neuroscience. Frontiers in Systems Neuroscience, 2:249, 2008.
Benjamin Lahner, Kshitij Dwivedi, Polina lamshchinina. Monika Graumann, Alex Lascelles, Gemma Roig, Alessandro Thomas Gifford, Bowen Pan, Young Jin, N Apurva Ratan Murty, et al. Modeling short visual events through the bold moments video fmri dataset and metadata. Nature Communications, 15(1);2241, 2024.
Yulia Lerner, Christopher Honey, Lauren J Silbert, and Uri Hasson. Topographic mapping of hierarchy of temporal receptive windows using a narrated story. Journal of Neuroscience, 31(8): 2906-2915, 201
13

---

Published as a conference paper at ICLR 2026
Kunchang Li, Xinhao Li, Yi Wang, Yinan He, Yali Wang, Limin Wang, and Yu Qiao. Videomamba: State space model for efficient video understanding. In European Conference on Computer Vision, pp. 237-255. Springer, 2024.
Jessica Loke, Noor Seijdel, Lukas Snoek, Lynn KA Sörensen, Ron van de Klundert, Matthew der Meer, Eva Quispel, Natalie Cappaert, and H Steven Scholte. Human visual cortex and deep convo- lutional neural network care deeply about object background. Journal of Cognitive Neuroscience, 36(3)5511-56, 2024.
Zejin Lu, Adrien Doerig, Victoria Bosch, Bas Krahmer, Daniel Kaiser, Radoslaw M Cichy, and Tim C Kietzmann. End-to-end topographic networks as models of cortical map formation and human visual behaviour. Nature Human Behaviour, pp. 1-17, 2025a.
Zejin Lu, Sushrut Thorat, Radoslaw M Cichy, and Tim C Kietzmann. Adopting a human develop- mental visual diet yields robust, shape-based ai vision. arXiv preprint 1.X1::2507.03168, 2025b,
Emalie McMahon, Elizabeth Jiwon Im, Michael Bonner, and Leyla Isik. A spatiotemporal hierarchy for social interaction perception in the lateral visual stream. OSF Preprints, 2025.
Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfreund, Carl Vondrick, et al. Moments in time dataset: one million videos for event understanding. IEEE Transactions on Pattern Analysis and Machine Intelligence, 5222))502-508 2019.
Mathew Monfort, Bowen Pan, Kandan Ramakrishnan, Alex Andonian, Barry A McNamara, Alex Lascelles, Quanfu Fan, Dan Gutfreund, Rogério Schmidt Feris, and Aude Oliva. Multi-moments in time: Learning and interpreting models for multi-action video understanding. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 44(12):9434 10449444444445,,,, 2021.
Omer Moussa, Dietrich Klakow, and Mariya Toneva. Improving semantic understanding in speech language models via brain-tuning In The Thirteenth International Conference on Learning Rep- resentations, 2025.
Lukas Muttenthaler, Klaus Greff, Frieda Born, Bernhard Spitzer, Simon Kornblith, Michael C Mozer, Klaus-Robert Müller, Thomas Unterthiner, and Andrew K Lampinen. Aligning machine and human visual representations across abstraction levels. Nature, 647(8089)3399 -55,, 2025.
Hamed Nili, Cai Wingfield, Alexander Walther, Li Su, William Marslen-Wilson, and Nikolaus Kriegeskorte. A toolbox for representational similarity analysis. PLoS Computational Biology, 10(4))e1003553, 2014.
Marc R Nuwer, Giancarlo Comi, Ronald Emerson, Anders Guérit, Hermann Hinrichs, Akio Ikeda, Fransisco Jose Luccas, and Peter Rappelsburger. Ifen standards for digital recording of clinical eeg. Electreencephalogaaphy and Clinical Neurophysi- ology, 106(3):259-2611 1998.
Jean-Michel
Pablo Oyarzo, Johannes JD Singer, Kohitij Kar, Diego Vidaurre, and Radoslaw M Cichy. Adaptive recruitment of cortex-wide recurrence for visual object recognition. bioRxiv, pp. 2025-10, 2025,
Brian E Russ, Kenji W Koyano, Julian Day-Cooney, Neda Perwez, and David A Leopold. Temporal continuity shapes visual responses of macaque face patch neurons. Neuron, 1 1106699933 2023.
Shahd Safarani, Ame Nix, Konstantin Willeke, Santiago Cadena, Kelli Restivo, George Denfield, Andreas Tolias, and Fabian Sinz. Towards robust vision by multi-task learning on monkey visual cortex. Advances in Neural Information Processing Systems, 34:739-751. 2021.
Christina Sartzetaki, Gemma Roig, Cees GM Snock, and Iris IA Groen. One hundred neural net- works and brains watching videos: Lessons from alignment. In The Thirteenth International Conference on Learning Representations, 2025.
Martin Schrimpf, Jonas Kubilius, Ha Hong. Najib J Majaj, Rishi Rajalingham, Elias B Issa, Ko- hitij Kar, Pouya Bashivan, Jonathan Prescott-Roy Roy. Franziska Geiger, et al. Brain-score: Which artificial neural network for object recognition is most brain-like? bioRxiv, pp. 407007, 2018.
14

---

Published as a conference paper at ICLR 2026
Noor Seijdel, Jessica Loke, Ron Van de Klundert, Matthew Van der Meer, Eva Quispel, Simon Van Gaal, Edward HF De Haan, and H Steven Scholte. On the necessity of recurrent processing during object recognition: it depends on the need for scene segmentation. Journal of Neuro- science, (1(9))6228 --289, 2021.
Thomas Serre, Aude Oliva, and Tomaso Poggio. A feedforward architecture accounts for rapid categorization. Proceedings of the National Academy of Sciences, 104(15):6424 6429, 2007.
Ilia Sucholutsky, Lukas Muttenthaler, Adrian Weller, Andi Peng, Andreea Bobu, Been Kim, Bradley C Love, Christopher J Cueva, Erin Grant, Iris IA Groen, Jascha Achterberg, Joshua B Tenenbaum, Katherine M Collins, Katherine Hermann, Kerem Oktar, Klaus Greff, Martin N Hebart, Nathan Cloos, Nikolaus Kriegeskorte, Nori Jacoby. Qiuyi Zhang, Raja Marjich, Robert Geirhos, Sherol Chen, Simon Korblithh, Sunayana Rane, Talia Konkle, Thomas O' O'Connell, Thomas Unterthiner, Andrew K Lampinen, Klaus Robert Muller, Mariya Toneva, and Thomas L. Griffiths. Getting aligned on representational alignment. Transactions on Machine Learning Research, 2025. ISSN 2835-8856. Survey Certification, Expert Certification.
Yingtian Tang, Abdulkadir Gokce, Khaled Jedoui Al-Karkari, Daniel Yamins, and Martin Schrimpf. Many-two-one: Diverse representations across visual pathways emerge from single objective. bioRxiv, pp. 2025 07, 2025.
Rufin VanRullen and Simon J Thoree The time course of visual processing: from early perception to deecision-makin Journal of Cognitive Neuroscience, 13(4):454-46) 2001.
Roel M Willems and Marius V Peelen. How context changes the neural basis of perception and language. IScience, 24(5), 2021.
Siying Xie, Daniel Kaiser, and Radoslaw M Cichy. Visual imagery and perception share neural representations in the alpha frequency band. Current Biology. 30(13):2621 (1113)3:6112622) 12627,, 2020.
Daniel LK Yamins, Ha Hong, Charles F Cadicu, Ethan Solomon, Darren Seibert, and James DiCarlo. Performance-oppimized hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences, 111(23)88619-88)) 8624, 2014.
Bolei Zhou, Agata Lapedriza, Aditya Khosla, Aude Oliva, and Antonio Torralba. Places: A 10 mil- lion image database for scene recognition. IEEE Transactions on Pattern Analysis and Machine Intelligence, 40x6):1452-1464 2017.
Lianghui Zhu, Bencheng Liao, Qian Zhang, Xinlong Wang. Wenyu I Liu, and Xinggang Wang. Vision mamba: Efficient visual representation learning with bidirectional state space model. Forty-first International Conference Machine Learning, 2024.
Marius Zimmermann, Angelika Lingnau, Katharina Frey Dochler, Lisa Kaiser, Alice Stephan, and Julian Wieland. The spatiotemporal neural dynamics of action-related features underlying action recognition. OSF Preprints, 2025.
15

---

Published as a conference paper at ICLR 2026
APPENDIX
A EEG DATASET DETAILS
The EEG Moments Dataset was collected by the Neural Dynamics of Visual Cognition group, at the Freie Universität Berlin. The dataset will be made public in the near future EEG data during natural video viewing was collected for 6 participants, all with normal or corrected to to-nommal vision. Procedures were approved by the ethical committee of the Department of Education and Psychology at Freie Universität Berlin and were accordance with the Declaration of Helsinki. Stimuli were the exact same in Lahner et al. (2224) 1102 3-second second videos, sampled from Monfort et al. 22119 2021). Stimuli were square-cropper and resized to 268x268 pixels (5xx50 visual angle).
Stimuli were divided in non-overlapping sets, "train set" of 1000 videos with 6 repeats and a "test set" of 102 videos with 24 repeats. The dataset is accompanied with crowd-sourced metadata, to an- notate each clip with five word-level scene, object, and action labels, sampled from Places365 (Zhou (et al. 20 THINGS (Hebart et al., [023). and Multi-Moments in Time (Monfort et al. (021) re- spectively. Additionally, provided were five sentence-level text descriptions, spoken transcript, memorability score and a memorability decay rate.
The experiment consisted of passive viewing paradigm with an orthogonal detection task have participants stay attentive, where participants had to report after X nr of random trials whether specific object/sceneeaction was present. The total experiment consisted of 8 sessions, that each contained 3 repeats of the 250 train videos and 3 repeats of the 102 test videos. Each session consisted of 16 runs of 66 trials, in between which participants could take breaks.
Every trial sequence consisted of Is of blank baseline screen, followed by 3s showing video, another 0.25s baseline and then of blink time. In the case of prompted trials the second baseline was followed by 3s prompt screen, which was then again followed by the 2s blink time. Stimuli were presented on a grey background with red fixation cross present during the whole experiment.
EEG data was recorded using 1 128-channel actiCAP set up with electrodees arranged according to the standard 10-10 system (Nuwer et al.) 1998) and Brainvision actiCHamp amplifier. Data was recorded at sampling rate of 1000 Hz with online filtering (between 0.1 Hz and 100 Hz) and rereferenced (to Fz). We performed offline preprocessing using Python and MNE (Gramfort et al., 2013). The continuous EEG data was epoched into trials from -0.2s to 3.5s with respect to stimulus onset and subsequently baseline corrected by subtracting the mean of the pre-stimulus interval for each trial and channel separately. The data was then temporally down- sampled to 50 Hz. Next, performed multivariate noise normalisation (MVNN) based on the covariance matrices of each time- point reweigh (un)reliable sensors and (de)emphasise specific spatial frequencies, as recom- mended for multivariate pattern analysis methods like RSA (Guggenmos et al. (018). We performed analyses on two separate sets of electrodes: posterior partition consisting of 35 electrodes and frontal partition consisting of 54 electrodes. The posterior partition contained the following elec- trodes: 'Pz', 'P3', 'P7', '01', 'Oz', '02', 'P4', 'P8', 'P1', 'P5', 'PO7', 'PO3', 'POz', 'PO4', 'PO8', 'P'', 'P2', 'p9', 'PPO9h', 'PO9', '09', 'O11h', 'PPO1h' The frontal partition contained the fol- lowing electrodes: 'Fpl', 'F3', 'F7', 'FT'', 'FC5', 'FCI' 'FTI0', 'FC6', 'FC2', 'F4', 'F8', 'Fp2', "FF7' 'AF3', 'AFz', 'F',, 'F5', 'FT7', 'FC3', 'FCz'. 'FC4', 'FT8', 'F6', 'F2', 'AF4", 'AF8' 'F9' AFFIhh, 'FFC1h', 'FFCSh', 'FT77h', 'FCC3h', 'FCC4h', 'FTT8h', 'FFC6h', 'FFC2h', 'AFF2h', "F10', 'AFpl', 'AFF5h', 'FFT9h', 'FF77h', 'FFC3h', 'FCCIh', 'FCC5h', 'FTT9h', 'FTT10h', "FCC6h', 'FCC2h', 'FFC4h', 'FFT8h', 'FFTIOh', 'AFF6h'.
For further inquiries or requests, contact Alessandro Gifford (alessandro (alcssandro.gifford@@ twttoooofffoo@@@gillcmmmmm
16

---

Published as a conference paper at ICLR 2026
B MODEL DETAILS
Table 1: Model families. Action recognition models are trained on Kinetics 400 [ay et al.
2017
those also available other datasets marked by a,b.
Image Object Recognition
Action Recognition
CNNs
Transformers
SSMs
CNNs
Transformers
SSMs
AlexNet
CAiT
VideoMambi
6 CSN
2
MViTv2"
8
VideoMannuu
NenseNet
TimeSformer
2.
EfficientNet
DEIT
R2P ID
2
Uniformer
Video
RegNet
MVTTv2
SlowFast
Unifarmer22
4
RosNet
Swin
4
Slow
VideoMAE
VideoMAEy2
VGG
VIT
TSM
3
VideoSwin"
WideRRsNet
X3D
Inception
RepVGG
Image
SeResNo(XOt
C2D
Tirmer
2
Xception
TSN
TSN
26
26+8
14+2
Availabilitty also Kineticx 710
Carrerr 2019
"Availability also on Something Something 42 Goyal 0t
2017
Table 2: Exhaustive account of all models.
Image Object Recognition
Action Recognition
SSMs
Transformers
AlexNot
CAiT_S
VidcoMMmbbLT
MVTTv2_5*
VidcoMaumb...NNI...00
DenscNet161
CAITXXS
VideoMamisu.8
IR.CSN.R152 INfrscnnJG65M
MVTTv2.B*
VideoMamba..TINI11
DenscNet201
ConViT_S
V
TimeSfomer DivST
VidcoMMabb..SJNNk....
EllickanNeBB33
ConViT_B
CSN R152 IG65M
JoinST
VidcoMamba. S.NNkammaaa.NNI....
EfficienNNeB66
DETT S
IP CSN R152 JG65M
Uniformer S
VideoMamba M INIk f16
RegNetX 16gf
DRIT_
IP_ IP_CSN_R152
Uniformer_.
VidenMtamha, M_inin,,,INNKK..
RogNetY
MVTTv2.S
Uniformerv2.BB
VideoMamba.
RcsNet34
MVTTv2_8
131)_R50_dopprd
Sonformerv2_B.A410000
TideoMamba,M_muss
ResNc:50
Swin. T
13D R50 embgauss
VidcoMAE B
ResNer101
Swin S
B3D R50, gauss
VideoMAEv2_S
ResNet152
Swin.R
[3D)_800_heay
MAEv2U8
ResNeX:50
Twins.pepv_B
R2PID_RS R_0
VideoSwin.T
ResNeX1101
VITSS
SlowFisi,R50
Vi
VGGL1
VidcoSwin
VGGLIBN
Slow R50
VGG19
Slow R101
VGG19BN
Slow RSO_IRS
WideMesNet50
Slow RSI.INNI...mbbass
WidcResNec101
TPN R50
InceptionV4
RepVGGa2
TSM_R50 _oopproddd
RepVGG82
TSM_R50.cmm R50..mgg.ebbsss
SeResNer50
TSM_RC RMM_R50,2 R._550_guusss
SeResNeX150
TSM. MobOne..44
Xception41
X3D_S
Xception71
CMD R50 no50nl
TimeSfommrr SpaceOnly TSN_Swin
C2D.R101 _nopool
C2D R50 pool8
C22D_R50_pool16
TSN R50*
TSN_RIOF
TSN D161
TSN. MobOne.s4
"Availability
al. 2019
Availability also on Something Something < RGoyaa 2017
17

---

Published as a conference paper at ICLR 2026
C SUPPLEMENTARY ANALYSES
Test - Posterior
sub
Decoding accuracy (0.5)
0.4
sub
sub
0.3
sub
0.2
sub
0.1
Train Posterior
Train Frontal
Decodin accuracy (40.))
0.3
0.1
0.0
Time (5)
Time (5)
Figure 6: One versus one stimulus identity decoding per subject, corrected by chance level (-0.5). Note: the decoding accuracy on the train set computed using randomly selected subset of videos, matching the number in the test set (n=102).
Posterior Electrodes
imae-inkk maae-ki000 viceo-8400
Frontal Electrodes
Stage
Stage II
Stage III
Stage IV
Stage
Stage
Rew
UNC opsted
Accuney Oh 44000
Accuary Fii Inik kebdy
Accumey th, witk k400)
Accurcy
Accuary Hi, Inlk
Accuncy
Ruge 555177
GELOPS
Modal GFLOPs
GPLDPs
Figure 7: (Top) Relationship of brain alignment with model accuracy (in ImageNet- 1k for object recognition models, and in Kinetics-400 for action recognition models). (Bottom) Relationship with model computational complexity (in GFLOPs). Only stages during which models showed significant brain alignment in Fig. 2 are shown.
18

---

Published as a conference paper at ICLR 2026
Posterior Electrodes
Image-keco
Frontal Electrodes
Stage! 500e
Stage
Stage
Stage IV
Stagel
Stage II
1000 2 2
sampligg
Model compling
comple
Model compling
Modell sampling 1600
Nonde compling
Assooeed 2 2008
Add
Auteea 2
Vodi contect wenokertaneel
Mode contect winsbertinnes
Mode contont Flones)
Moddd contost window yiarso)
Vodi contect weckerthameil
Mock Jicnen
Figure 8: (Top) Relationship of brain alignment with model sampling rate (fps). (Bottom) Rela- tionship with model context window (frames). Only stages during which models showed significant brain alignment
Fig. are shown. Here, model sampling rate and context window positively correlate with the brain alignment score of temporally integrating models (video-k400) significantly (p<c0.05) in posterior electrodes, stages and IV.
sub
sub 2
image-ndi Imagelo120 vede--et0
sub 3
sub
sub 5
sub
Figure 9: Subject level brain alignment results for the test set for posterior and frontal electrode partitions.
19

---

Published as a conference paper at ICLR 2026
0.
Wideloug Chi Thrrsommer
Modedrupp
ONN
no pre
04
Terolomers
03
02
MecetSrop
pre- pre-iman im-immae
Tranelvmee
Fandomar
RMR
05
Vedet Chil Transfomen
WecalSoup
clea Tarsfomers
pre- pre-vided video
04
pre- pre-video-ssi
Tarsfomers
nMa
04
0.
III
IV
IV
Figure 10: Additional control for pretraining in the architecture comparison of Fig.
Modeffsp DIes
Tarstomers Ree
0.6
Transforerss
0.0
0.5
1.0
2.0
25
3.0
3.5
0.5
3.0
2.0
7.5
3.0
3.5
Figure 11: Showing the layer plot of Fig.
4b for the full timecourse.
Transfomers
IV
IV
Figure 12: The comparison of Fig.
only within object recognition models.
20

---

Published as a conference paper at ICLR 2026
Transformers
05
04
prrimage prteddo prestleboal
premmage provilee
provicec-ao
0.2
0.0
Hode Group
NocalGroa
napro
promagg
preirage
CNNS
prouidenet
SSMs
I1
IV
III
IV
Figure 13: Additional control for architecture in the pretraining comparison of Fig.
5
MocelGroup 5th
ModetGroup
5th
0.4
Voluo
0.0
I
II
III
IV
III
IV
Figure 14: No significant differences between fine-tuning datasets 1Sth-Sth-v2 vs. K400).
0.5
5pearmans tho
0.5
Sahamenns
0.4
Kinetics-710 Kinetics 400
0.4
0.3
0.1
0.0
(seconds)
(ecconss)
Figure 15: No significant differences between fine-tuning datasets (K710 vs. K400).
Spp
00.05
0.5
Figure 16: Comparing VideoMamba versions; object (blue), prrtrtt-ret action (orange), self- supervised-pretrainee action recognition (green), and pure self-supervised (red).
21

---

Figure 18: Non noise-scaled brain alignment of time-aggregated model features (RSA instead of Published as a conference paper at ICLR 2026
posterior
frontal
2.5
9.5
0.4
0.3
0.2
0.3
-51
Figure 17: Same as Fig. but without scaling to the upper noise ceiling. Gray shading shows both lower and upper noise ceilings.
posterior
frontal
0.6
0.4
0.0
CT-RSA). Gray shading portrays the noise ceilings.
D USE OF LARGE LANGUAGE MODELS (LLMs)
In the writing of this paper, LLMs were only minimally used aid polish writing, such as asking for synonyms or rephrasing at the sentence or sub-sentence level.
22


Reviews:
[{"review_id": "dhJ7V7ieUx", "reviewer": "Reviewer_uQvM", "summary": "This paper presents the first large-scale representational alignment benchmarking study between human EEG recordings during natural video viewing and deep neural networks. They benchmark 110 architectures, including imaging models, video models, and state-space model and using a Cross-Temporal Representational Similarity Analysis (CT-RSA), that extends the classic RSA analysis along the temporal dimension. It works by building representational dissimilarity matrices (RDMs) for each EEG timepoints and for each model layer and timepoints across all videos, correlating them to form a cross-temporal map of brain-model correspondence. By extending RSA into the temporal domain, the method aims to reveal when and how neural network representations align with evolving brain activity during continuous visual perception.", "questions": "- The alignment of CNNs with frontal EEG activity seems counterintuitive; could the authors expand on this? \n\n- How does correlation metrics evolve with relative models performance of image and video models on there respective benchmarks? Related to that, could the authors clarify how self-supervised models were selected and whether their pretraining objectives influence alignment differently than supervised models?\n\n- How robust are the results to variations in video duration or temporal resolution - for example, would similar dynamics appear with longer or continuous movie stimuli? what do you expect? \n\n- How sensitive is CT-RSA to differences in model frame sampling rates? what would happen if video models were trained on the same sampling rate as the EEG signal is processed? \n\n- Could CT-RSA be extended to multimodal representations (audio-visual or language-conditioned models) to explore cross-modal dynamics in brain alignment? One could think of movie-watching experiments (mostly with fMRI) for this kind of studies as some aspects of cognition can only be unlocked from longer stimuli, especially with complex stimuli such as audio-visual. \n\n- It is not a strong opinion, but the title of the paper might be misleading with respect to the reference to \"Mixture of Experts\" (MoEs). The “mixture of experts” framing is conceptually interesting but remains speculative; the authors do not provide evidence of explicit modular switching mechanisms.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The CT-RSA framework is an elegant methodology, allowing dynamic model-brain alignment to be explored at high temporal sampling rate using EEG datasets. The paper contributes to bridging fMRI-based alignment work (limited by slow dynamics in general) and fast EEG dynamics, offering complementary insights into temporal processing. The framework is generalisable and could serve as a benchmark pipeline for cross-modal studies comparing other neural modalities (MEG, ECoG, fMRI)\n\nThe paper is globally well-written paper, with thorough contextualisation (related work section), and an extensive discussion sections. \n\nThe study covers an unusually wide model space (over 110 architectures).\n\nThe results uncover a meaningful temporal hierarchy: posterior EEG activity transitions from low-level visual encoding to mid-level integrative representations, while frontal activity remains stable and semantically oriented.\n\nInteresting exploratory results between frontal and posterior electrodes encoding preference w.r.t various types of models (image, video) and training schemes (pre-training, finetuning only)", "weaknesses": "Despite the large-scale benchmarking, it looks like the neuroscientific interpretation remains underdeveloped/underexploited: \n- The paper provides little theoretical discussion linking the observed dynamics to known cortical hierarchies or cognitive models of video perception.\n- It is not clear why this segregation (posterior/frontal) of electrodes was chosen. Was other cluster of electrodes tested? What is the limit of this choice? \n\nRegarding the methods, one could highlight a few blind spots in the methodology: \n- The temporal stages (I-IV) are defined by visual inspection rather than a formal statistical segmentation or theoretical motivations, which weakens their interpretability.\n- While CT-RSA is a strong method, the heavy reliance on correlation-based RSA prevents causal or mechanistic insights; additional decoding analyses could strengthen the conclusions.\n- The methodology does not account for the spatial-temporal organisation of brain activity or the existence of functional networks or prior anatomical organisation; instead, it effectively collapses the EEG pattern at each time step into a single similarity value, losing information about spatial interactions and network-level dynamics.\n- It would have been interesting to evaluate differences between subjects (if any?) or mentioned it otherwise. \n\nResults: \n- The paper could benefit from a clearer distinction between what is novel and what is confirmatory from the previous fMRI work (Sartzetaki et al. (2025)).\n- It is difficult to draw any clear conclusions regarding the results of different type of DL architecture and training schemes, and how it compares to previous work with fMRI.  \n\nLimitations of the methods: \n- There is not enough discussion of the limits of using EEG for this kind of study (spatial resolution), despite the advantage of the temporal resolution. How do you approach this tradeoff? \n- Some part could benefit from more clarity: \"Responses in posterior electrodes, after initial alignment to hierarchical static object processing, best align to mid-level representations of temporally-integrative actions and closely match the unfolding video content.\"\n- Figure 1 could be more clear.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "p00HG5XQpF", "reviewer": "Reviewer_3fFP", "summary": "This study compares deep learning models to brain activity (EEG) and finds that the brain processes videos like a \"dynamic mixture of experts.\" Different brain regions align with different model typesposterior areas with mid-level dynamic features and frontal areas with high-level static actions. The findings suggest that a single, ideal AI model would need to flexibly combine these capabilities.", "questions": "Please see the Weaknesses section.", "limitations": "", "rating": 4, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper is clearly written and well-organized, making it reader-friendly.\n\n2. The article conducted a large number of rigorous experiments and collected new EEG data to explore this issue.\n\n3. The article provides a reasonable explanation for all the results obtained from the experiment, and it is consistent with some existing conclusions in the field of neuroscience and model explanations.", "weaknesses": "1. From several articles about brain and model alignment (not limited to video understanding), they all stated that aligning research can help in designing better deep learning models. However, no improved models have been created based on these works. Each time a new model is developed, alignment and explanation are made based on it (e.g., Mixture of Experts, MoE has been used for several years, not based on this article). Even some brain activity explanations of BERT [1] have now been considered outdated by engineering standards. Could the author provide me with some examples of recent studies on brain activity that actually guide the design of deep learning models?\n\n2. It is uncertain how reliable the research on EEG signal alignment is. In my view, EEG signals are very blurry and unstable. They contain a lot of noise, and even for the same subject, EEG responses to the same stimulus can vary when collected multiple times. Did the author first verify the reliability of the collected EEG signals? For example, conducting multiple collections to verify consistency, etc.\n\n3. Furthermore, all the video clips of the experimental studies were only 3 seconds long. It is completely fine to explore the basic visual responses or decode visual signals within this length. But can we really explore the complex reactions of the brain within this time window? (For example, video understanding) Some of the conclusions in the article may have little connection with video understanding. For instance, the initial brain activity of each video segment might merely indicate that we have just viewed a new video. Among the fleeting content, this article not only studies comprehension but also examines brain activity and the changes in the model over time. I'm not sure if this is reliable.\n\n4. Furthermore, current research mainly focuses on alignment, meaning which time points show changes in brain activity. However, we do not understand what these changes actually represent. Although the author provided an explanation, it does not serve as proof. In other words, these changes can be interpreted in various ways, and there is no way to tell which interpretation is correct. It is also unreasonable (no evidence) to assume that the model should match the EEG (brain) activity. In other words, we neither understand how the brain works nor grasp why the model functions. So, does alignment make any sense in this situation?\n\n[1] BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding\n\n\nDeclaration: I lack profound background knowledge of brain-model alignment. But I have interests and questions for this field. If the questions raised are unreasonable or if there are common limitations in the field, the author can provide an explanation. Also, please ACs consider the weight of my questions as appropriate.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "EVTIzXqflM", "reviewer": "Reviewer_mriQ", "summary": "The paper presents the first large-scale benchmarking of over 100 deep neural networks against EEG recordings collected during natural video viewing. Using a novel Cross-Temporal Representational Similarity Analysis (CT-RSA), the authors analyze alignment dynamics between model features and brain responses. Results reveal distinct temporal processing stages in posterior and frontal electrodes, suggesting that brain activity during video perception reflects a dynamic mixture of expert models tuned to different representational demands.", "questions": "See above.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Interesting large-scale study linking deep video models with dynamic EEG signals.\n* Introduction of CT-RSA as a principled extension of RSA to capture cross-temporal alignment.\n* Rich, interpretable findings showing stage-wise specialization in brain responses and model correspondences.\n* Strong relevance to both neuroscience and AI, offering biologically inspired insights for model design.", "weaknesses": "* While conceptually interesting, the empirical novelty beyond previous fMRI-based benchmarks feels incremental, focusing mainly on EEG substitution.\n* The EEG dataset is limited to six subjects, which may restrict generalizability.\n* Some claims about \"mixture of experts\" remain metaphorical rather than computationally substantiated.\n* The paper does not analyze the robustness of CT-RSA to preprocessing or electrode partitioning choices, nor quantify its improvement over standard RSA or fMRI-based alignment methods, leaving the methodological advantage insufficiently demonstrated.\n* To what extent do the findings generalize across individuals, given the small sample size?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "kAWwjDwwyM", "reviewer": "Reviewer_fzAq", "summary": "This paper presents the first benchmark for the alignment between artificial image / video representations from neural networks and dynamic brain responses to natural videos, recorded using EEG.  This benchmark is carried out on a new dataset EEGMD (not public yet) that extends the Bold Moments Dataset (BMD) to EEG responses (to natural videos). To compare between models, the authors introduce a time-aware extension of the RSA method: Cross-Temporal Representational Similarity Analysis (CT-RSA), that allows computing dissimilarity between time-courses of EEG and model representational dissimilarity matrices. This allows them to make a number of significant findings about how the representations in artificial models and brains correlate across time and brain regions.", "questions": "- Apart from the already presented axis of analysis, could the authors present results correlating alignment with task-related performance of video / image embedding models on a given task ?", "limitations": "", "rating": 4, "confidence": 1, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well written, easy to follow, the methodology and metrics are clearly defined\n- The assumption resulting from the analysis of the paper, that is the brain as a dynamic mixture of experts, is indeed supported by the results of the benchmark across the different architectures, time-aware / static models, model types, ...\n- There are a number of conclusions that are new to the best of my knowledge (e.g. 'frontal activity being best capture by early static semantic action representations')", "weaknesses": "- The measurement of alignment is rather complex (UNC-scaled Spearman's rho) and the paper misses a discussion of why using this metric is superior to much simpler methods (e.g. just using Pearson's r across time). Since cluster analysis is absent, the advantage of using RDMs is not obvious.\n- The paper makes a number of interesting findings but does not seem to delve into their interpretations (e.g. SSMs better aligning to posterior processing)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)