One Hundred Neural Networks and Brains Watching Videos: Lessons from Alignment
Authors: Christina Sartzetaki
Gemma Roig
Cees G. M. Snoek
Iris Groen
Venue: ICLR 2025 Poster (2025)
Keywords: representational alignment
Representational Similarity Analysis
RSA
benchmarking
neuro-AI
video AI
neuroscience
fMRI
cognitive AI
Primary area: applications to neuroscience & cognitive science
TL;DR: We benchmark 99 image and video models on brain representational alignment to fMRI data of humans watching video.

Abstract: What can we learn from comparing video models to human brains, arguably the most efficient and effective video processing systems in existence? Our work takes a step towards answering this question by performing the first large-scale benchmarking of deep video models on representational alignment to the human brain, using publicly available models and a recently released video brain imaging (fMRI) dataset. We disentangle four factors of variation in the models (temporal modeling, classification task, architecture, and training dataset) that affect alignment to the brain, which we measure by conducting Representational Similarity Analysis across multiple brain regions and model layers. We show that temporal modeling is key for alignment to brain regions involved in early visual processing, while a relevant classification task is key for alignment to higher-level regions. Moreover, we identify clear differences between the brain scoring patterns across layers of CNNs and Transformers, and reveal how training dataset biases transfer to alignment with functionally selective brain areas. Additionally, we uncover a negative correlation of computational complexity to brain alignment. Measuring a total of 99 neural networks and 10 human brains watching videos, we aim to forge a path that widens our understanding of temporal and semantic video representations in brains and machines, ideally leading towards more efficient video models and more mechanistic explanations of processing in the human brain.

Published as a conference paper at ICLR 2025
ONE HUNDRED NEURAL NETWORKS AND BRAINS WATCHING VIDEOS: LESSONS FROM ALIGNMENT
Christina Sartzetaki', Gemma Roig", Cees G.M. Snoek & Iris L.A. Groen Informmtics Institute, University of Amsterdam, The Netherlands Department of Computer Science, Goethe University Frankfurt, Germany fc. sa.satzztakk,, L.i.a. wrw.a.groonneeeo..o.oooo.o g @uva.
ABSTRACT
What can learn from comparing video models to human brains, arguably the most efficient and effective video processing systems in existence? Our work takes step towards answering this question by performing the first large-scale benchmarking of deep video models on representational alignment to the human brain, using publicly available models and recently released video brain imaging (fMRI) dataset. We disentangle four factors of variation in the models (temporal modeling, classification task. architecture, and training dataset) that affect align- ment to the brain, which we measure by conducting Representational Similarity Analysis across multiple brain regions and model layers. We show that temporal modeling is key for alignment to brain regions involved in early visual process- ing. while a relevant classification task is key for alignment to higher-level regions. Moreover, we identify clear differences between the brain scoring patterns across layers of CNNs and Transformers, and reveal how training dataset biases transfer to alignment with functionally selective brain areas. Additionally, negative correlation of computational complexity to brain alignment. Mcasuring a total of 99 neural networks and 10 human brains watching videos, we aim to forge a path that widens our understanding of temporal and semantic video representa- tions in brains and machines, ideally leading towards more efficient video models and more mechanistic explanations of processing in the human brain.
uncover
INTRODUCTION
Humans are extremely efficient in processing the constant streams of visual information they re- ceive, relying on motion and temporal information on top of visual semantics to understand their environment (Sekuler et 2002). How current state-of-the- video models compare to that stan- dard is question that is often addressed by comparing their performance to human baselines (Zhou) al. [018aa] Andonian et al., 2020), but is much more under-explored with respect internal rep- resentations. Representational alignment was defined in (Sucholutsky et al. (2023) as "the extent to which the internal representations of two or more information processing systems agree". is a comerrsonn of cognitive computational neuroscience (Kriegeskorte & Douglas 2018). discipline that aims to identify neural mechanisms underlying cognition by employing task musk-performing com- putational models, such deep neural networks, for hypothesis testing. This has been shown to be a highly progressive research program yielding novel neuroscientific insights (Doerig et al.) 2023). Identifying model design choices that strongly impact alignment can not only shed light on the un- derlying brain mechanisms, but also guide machine learning on borrowing brain's 's advantages such as efficiency and robustness, for example using brain-aligned designs as starting points for further model development (Lu et al.) 2024b). Our work aims to fill the gap in exploring the representa- tional alignment of deep video models to the human brain, joining and increasing understanding of temporal modeling in brains and machines, and pointing towards more efficient video models.
Human visual information processing has classically been separated into two separate streams, ven- tral and dorsal (Mishkin et al. 1983). The ventral stream is thought to process more static informa- tion and the dorsal stream more dynamic information (Kravitz et al., 2011), whereby a motion area MT (Culham et al.] 2001) has also been identified. More recent work found evidence for third vi- sual pathway specializing in dynamic social perception (Pitcher & Ungerleider 2021 Küçük et al. 2024). Another organizing principle that has been identified in human visual processing is hier- archy of temporal receptive fields, with temporal integration extending over longer time-scales in
higher compared lower visual areas (Hasson al. 2008 Zhou al. 2018h] Groen et al. 2022

---

Published as a conference paper at ICLR 2025
10 subjects
B, Disaiiiarlty by mabis for Region of aubject
Brain Regions
Voxd voctors 9g
10 rapettions per video
Brain Representational Alignment
Target Region
B ((
of subject
Layer oot NN
M
R
1
M-1
Σ PDD M)
CNNs
99 Neural Networks
Feature voctoris ft
DenseNet
C2D
X3D
Transfomees
Xception
SlowFast Fast
Swin
MVIT
Tomporal modelling
TSM
TSN Uniformer
Dissimiartt moine layor
VIT
trained an
VideoMAE
Classi Dlcatification taok
Kinatics 400
Architesture
Kinetica 710
Someehing-Somettinng-
fastors of veriston
Training
datasal
Figure I: Overview of our pipeline for benchmarking video models on brain representational align- ment. We systematically compare 99 neural networks across 4 factors of variation on their alignment with fine-grained brain regions measured with Representational Similarity Analysis.
Brands et al. 2024). These neuroscience findings collectively suggest the presence of specialized mechanisms for motion and temporal information processing in the human brain.
In video AI, specifically machine learning, temporal information is usually modeled third input dimension, using mechanisms such as 3D convolutions (Tran et al. 2015) Carrriiaa & Zisserman 2017]; however there are only limited similarities with processing the brain, the effective temporal receptive fields in these networks usually stay constant throughout processing layers. Older models handling RGB images and optical flow as separate streams (Simonyan & Zisserman 2014) bear some resemblance to dual-stream processing in the brain, but most successful designs fall far from direct comparison (Feichtenhoter al. 2019 Lin et al. 2019). Recent papers focus on video Transformers (Liu et al., 2022; Wang et al. 2023b). including hybrids to combine benefits of CNNs and Transformers (local structure and long term dependencies) in temporal modeling (Li et al. 2022] 2023ap. Comparing the variety in engineering solutions from video-Al to the way humans process video can help adjudicate between models terms of their ability to capture the temporal aspects of information processing in the brain and increase understanding the representations they compute.
A line of work has recently set path in large-scale systematic benchmarking of models on represen- tational alignment, to both the human brain and behavior (Conwell et al. 2022 2024 Muttenthaler et al. 2023), , advancing the understanding of how these models internally compare to humans on common basis. However, these works have far been limited to the domain of static images; comprehensive analysis of video model alignment with human video processing is still lacking.
Our contributions:
We perform the first large scale benchmarking of video models on human brain representational alignment, recent publicly released video fMRI dataset. Specifically, consider 47 video models for action recognition and 41+11 image models for object and action recognition.
We decouple the alignment effects of temporal modeling from those of action space optimization by adding image action recognition models as control, as well as examine the impact of model architecture and training dataset, all comparing across fine-grained variety of brain regions.
We show that temporal modeling is the important factor for representational alignment to early brain regions, while action space optimization for alignment to late brain regions. We find distinct patterns emerging in different processing stages of CNNs and Transformers. along with an effect of training biases on alignment to functionally selective areas.
We report significant negative correlation of model FLOPs to alignment in several high-level brain areas, indicating that computationally efficient neural networks can potentially produce more human-like semantic representations.
2

---

Published as a conference paper at ICLR 2025
2 RELATED WORK
Large scale brain datasets. Compared to artificial intelligence, the field of computational cognitive neuroscience has been exposed to the concept of benchmarking only recently (Schrimpf et al. 2020),
with the majority of prior works consisting of stand-alone assessments of representational alignment on their own brain data for one only a few models. Works such as BrainScore (Schrimpf al.)
2018) and the Algonauts challenges (Cichy et al. 2019 | Gifford et al. 2023) have set the stage for representational alignment research on benchmark datasets where any model can be compared and comparisons can be widely accessed by everyone in the community, generating new insights via
data-driven exploration and field- meld-mg Datasets such as NSD (Allen et al. 2022) and THINGS (Hebart et al. 2023) have been established as benchmark image stimuli brain imaging datasets due to their size and high signal quality. For video stimuli there has also been some recent development
(Dima et al. 2022 Zhou et al. 2023 McMahon et al. (202), that is mostly focused on social actions. Most recently the Bold Moments Dataset (BMD) (Lahner et al. 2024) was introduced, putting forth scage,scale, highly reliable video fMRI dataset with extensive quality control to ensure suitability for Al model comparisons. In this work, we conduct the first extensive benchmarking of a total of 99 models on representational alignment with the fMRI data in BMD.
Measures of model-brain alignment. The two most widely used approaches for measuring brain- model alignment are voxel-wise encoding (Güçlü & Van Gerven 2015) and Representational Simi- larity Analysis (RSA) (Kriegeskorte et al., 2008). The former involves training linear regression to predict voxel space from a model's feature space and conduct all analyses there, modeling each voxel response independently rather than how voxels respond together. The latter projects both spaces into a third common space of pairwise condition patterss; it works by computing a distance metric be- tween the representations of each pair of test conditions for both voxel and feature space, forming the Representational Dissimilarity Matrix (RDM), which reflects differences between patterns of activity within populations of voxels. There are several influential works assessing model-brain
alignment for static images, using both voxel-wise encoding (Eickenberg et al. 2017: St- Yvey & Naselaris [2018] La Tour et al. 2022) and RSA (Cichy et al. 2016 Dobs et al. 2019 King et al. 2019|Bartnik et al. 2024) as well as methods combining the two approaches (Dwivedi et al. 202T
al.] 2022 2024). Many of these works validate the image models' alignment to the brain's s early-to-late processing hierarchy, as has also been shown with speech models (Millet al. 2022), but it has not been seen if a similar mapping can be found in video models. In this work, we use RSA to derive the representational alignment between the video features of each neural network's layers and the voxels in each brain Region of Interest (ROI) in BMD, contrasting this alignment between early and late stages of visual processing.
Konkle & Alvarez 2022 Conwell
Model-brain alignment benchmarking. Extensive benchmarking in the image domain was con- ducted by Conwell et al." 2002 2024) on NSD, who decoupled multiple factors that give rise to differences between image models, specifically model architecture, diet (training dataset), and train- ing task. They varied these one at a time while keeping all others constant, to assess which factors most influence model-brain alignment. A similar approach was followed in Wang et al. (2023a) to examine the effect of contrastive learning on model-brain alignment. There are also works exten- sively benchmarking neural networks on alignment to human behavior, using behavioral similarity judgment data (Muttenthaler et al. 2023 Marjich et al. 2023), where the latter also includes three video action recognition models. In Mineault et al. (2021). a limited testing of five video action recognition models was performed to compare against their own model trained with custom objec- tive. More recently, a study by Garcia et al. (2024) focused on social-action videos and conducted first, yet limited, video benchmarking by testing eight video models against 200 image models. In this work, we include 47 video models for action class recognition and 41+1 image models for object and action class recognition. Inspired by Conwell et al. (2022). we are interested in disentan- gling the role of different factors, and thus we vary the models' architecture by contrasting CNNs to Transformers, and the models' training dataset by contrasting three different datasets. Impor- tantly, we control for the change in task when comparing object recognition image models to action recognition video models, through varying one factor at a time by further comparing with image models trained for action class recognition. In addition to Conwell et al. (2022), apart from involy- ing dynamic stimuli, our analyses also address specific brain regions instead of the visual cortex as whole. In contrast to Garcia et al. (2024), we compare equally large number of video models to image models, on top of the important control of the training task and the comparison between different architectures and training dataset, and use RSA instead of voxel-wise encoding.
3

---

Published as a conference paper at ICLR 2025
3 METHODOLOGY
In Figure ! show overview of our methodology for measuring alignment of video models to the human brain | In the next three sections, we describe the design choices of the alignment measure, the video models, and the brain imaging dataset.
3.1 ALIGNMENT BY REPRESENTATIONAL SIMILARITY ANALYSIS
We start by motivating our choice of metric to measure the alignment between a neural network and a brain region. To observe patterns of activity that consider the interactions between groups of voxels (multívariate analysis), either Representational Similarity Analysis (RSA) (Kriegeskorte
al. 2008 combined approach that performs RSA on top of voxel-wise encoding (mixedRSA
or veRSA (Khaligh- Razavi et al., 2017) are good candidates. Conwell et al. (2022 2024) exten-
sively compared these methods and showed that veRSA often fails to uncover differences between models in terms of brain alignment. This is likely due to the fact that voxel-wise encoding allows for re- re-weighting of the model activations while mapping them to the brain, thereby optimizing brain predictivity but obscuring how the model representations correlate with brain responses out of the box. Based on our intent to benchmark the emergent alignment of Al models, we choose RSA P as stricter metric that can uncover more potential differences between models.
Representational Dissimilarity Matrix (RDM) computation. For brain Region of Interest (ROI) of human subject with voxel vectors and model layer/ / with feature vectors fi, the brain RDM (B,) and model RDM (M)) are calculated as follows:
B
r(u vg), M = 1- r-r
vi,j(i 0 osj <,
(I)
where N is the total number of videos and is the Pearson correlation, resulting in RDMs that symmetric with size - - 1)/2. To obtain Us we first average the voxels across measurements from all K repetitions of a stimulus video i, SO v/ = d Lk=0 U2 To create fi we first reduce the dimensionality of the original features to 100 Principal Components using Principal Compo- nent Analysis (PCA). We provide comparison with Sparse Random Projection (SRP) and the full dimensions in Figure 16. Appendix D No standardization is performed on the voxel and feature vectors, neither across videos nor across features to avoid distortions (Walther et al. 2016).
Correlation of RDMs. To derive the alignment of model layer and brain ROI, we correlate each layer RDM of each model (Mi) with each subject RDM each ROI (B.) using Spearman correlation (p), and then average the correlations across subjects for each model layer:
M-1 R
p(B,, Mi),
M
where M is the total number of subjects. In analyses where we show only one correlation value for the alignment of a whole model and a brain ROI, this is computed by R mari(Bi). i.c., the highest-correlating layer in the model.
Noise ceiling computation. Because of individual subject variability in brain data, noise ceilings for each ROI are computed to compare model RSA scores against the maximum obtainable score given the inter-subject variability (Nili et al. 2014). For the lower noise ceiling (LNC) we compute mean RDM across all subjects except one. Then we take the Spearman correlation of the left-out subject RDM and mean RDM, repeating for all the subjects and calculating the average. For the upper noise ceiling (UNC) we take the mean of all RDMs without removing subjects, compute the Spearman correlation of each subject RDM with the mean RDM, and average.
M-1 LNC =
M-1
M-1
M-1
Σ Pl
B, B, B,). UNC = M
B
B₁)
(3)
M
M
M
1=0
i0,,i
j=0
i=0
The upper noise ceiling signifies perfect correlation for the amount of noise in this ROI's data, often referred to as the maximum amount of variance that can be explained.
Due licensing restrictions the stimuli used in the hrain dataset, video frames shown the figure are sourced from representative videos captured by the authors themselves and are not subject to copyright. We utilize the RSA implementation from the Net2Brain python library (Bersch et al. 2022
4

---

Published as a conference paper at ICLR 2025
Table 1: Model families benchmarked. We sample the same number of CNN and Transformer image object recognition models as in our maximal set of video models. We also test 10 image models trained on action recognition; this division shown with the grouping on the right. Action recognition models are on Kinetics 400; those also available on other datasets are marked by a.b.
Image Object Recognition
Action Recognition
CNNs
Transformers
CNNs
Transformers
AlexNet
2
CAÎT
CSN
MVTTv24
2
DenseNet
2
ConViT
5
I3D
2 Tirmerormer
2
EfficientNet
2
DETT
R2P1D
Uniformer
Video
RegNet
MViTv2
SlowFast
Uniformerv2"
ResNet
Swin
Slow"
VideoMAE
ResNeXi
Twins
TaNet
VideoMAEv2
4
VGG
VIT
TPN
VideoSwin"
WideResNet
TSM
2
Inception
X3D
2
RepVGG
Image
2
SeResNe(X)t
4
C2D
Tiomerrrmer TSN
2
Xception
4
TSN
27
14
27+8
14+2
2019 Availability also on Something Something v2 Gnyal ct al.
''vailabiliity also Kinetics 710 Carreira et al
2017
Statistical significance. To test if model RDMs correlated significantly with brain RDMs, we per- formed permutation tests (Nili et al. 2014). For each model we permute the rows of all layer RDMs 1000 times using the same 1000 random permutations for all models and layers. We then calcu- late a null distribution by computing the Spearman correlations of all permuted RDMs with each subject RDM and average the resulting null distributions across subjects, separately for each model layer. For significance of a group of models against zero, we perform a two-tailed sign test between the median null distribution of all models in the group and the median observed Spearman correla- tion. To test for significant differences between two groups of models, we perform a two-tailed sign test between the null distribution created from the across across-group differences in the within-group group me- dian distributions, and the observed difference in the medians of the two model groups' Spearman correlations. We correct for multiple comparisons between model groups by applying Bonferroni correction equal to the number of group pairwise combinations.
3.2
VIDEO MODELS
Our goal was to benchmark as many publicly released video models as possible, sampling from different architectures (e.g. CNNs, Transformers) as equally as possible, while differentiating the effects of optimizing for the action classification task and temporal modeling on brain alignment.
Model choice. In total, we benchmark 99 models; of these, 41 are image models trained for object recognition on ImageNet, 10 are image models trained for action recognition on Kinetics 400 (Kay ct al. 70, and 41 are video models trained for action recognition on Kinetics 400. The distinction between the last two is made based on whether the models treat time in a non-trivial way, where triv- ial considered anything that makes completely separate computations per frame and only averages frame features before classification, aggregates frames with static pooling operations different stages. The remaining models are trained for action recognition on other datasets, namely Kinetics 710 (Carreira al. 2019) and
Seggiiinn--Smeehingg.... (Goyal al. (017). Image models trained on object recognition were ported from torchvisior? and timm while image and video models trained on action recognition were ported from m The main families of models used are listed in Table [ while a full list of all the model versions used can be found in Appendix B
Model feature extraction. We perform preprocessing according the functions provided by the models' sources (torchvision, timm, or mmaction2). Regarding the temporal dimension, for image models trained on ImageNet we perform inference for all frames and then average the features, while action recognition image models were trained on sequences of frames as input samples (only
html
5

---

Published as a conference paper at ICLR 2025
to aggregate with trivial pooling operations), for those we perform inference video inputs, as we do for video models. Because each model expects a specific length of timepoints sampled at a specific rate, different size batches of sub-clips are created per video and model, and the resulting sub-clip features are averaged for the whole video. We extract features from all higher-level blocks in the models (e.g., in a ResNet-type model of five blocks with four layers each, we extract features at the end of each block) and also include the final fully connected classification layer. We flatten the features after extraction, producing a single one-dimensional feature vector per layer.
3.3 BRAIN DATASET
We use the Bold Moments Dataset (BMD) (Lahner et al. 2024) consisting of whole-brain 3T fMRI
recordings (2.5 x 2.5 x 2.5 mm voxels, resampled TR of Is) from 10 subjects watching 1102 3s videos from the Moments in Time (Monfort al. 2019) and Multi-Moments in Time (Monfort et al. 2021 video datasets. Extensive details on the fMRI data acquisition can be found in Lahner et al. 2024); summary is provided in Appendix [ For each subject, 1000 videos were shown for 3 repetitions and those recordings make up the "training set", whereas 102 videos were shown for 10 repetitions and make up the "test set". In our analysis, we only use the 102 videos of the test set, whose high number of repetitions allows for the application of RSA. We report partial results on the 1000 video training set in Figure [7] Appendix [D] showing noise increase from insufficient repetitions. The test set videos are sensibly representative of the videos present in the whole dataset. Specifically, roughly 25% of the videos are actions of children and 22% actions of animals, another 22% people doing sports, 10% scenes with motion but no visible humans or animals (e.g. waterfall, a car), 6% people cooking, and 6% people performing some other manual labor.
Preprocessing and Regions of Interest (ROIs). We used preprocessed data provided by (Lahner et al. 2024). summary of the most important preprocessing steps, including anatomical alignment and ROI definitions, is again provided in Appendix A From the available brain ROIs, we first joined the two hemispheres by concatenating the voxels corresponding to each pair of areas, and then also joined the dorsal and ventral parts of V1 and V2. We group ROIs based on their anatomical location on the brain (Grill-Spector & Malach (004), in four groups: Early Visual Cortex (includes areas VI. V2. V3v. V3d), Ventral-Occiipita Stream V4, OFA, LOC, FFA, PPA), Dorsal Stream (V3ab, IPSO, IPS1-3, RSC), and Lateral Stream (EBA, TOS, MT, STS). Areas in Early Visual Cortex are considered early areas of processing, hV4 and V3ab are considered intermediate, and the rest are considered late areas of visual processing, based on the flow of information through the brain.
4 RESULTS
Differences in brain alignment between 99 neural networks processing videos could be related to number of factors. In the following sections we systematically examine the following factors: temporal modeling, classification task, architecture design, and training dataset. Finally, we also investigate how brain alignment relates to the models' computational complexity.
Video vs. Image models, controlling for the classification task. First, we compare models by varying the temporal modeling (Video vs. Image) and controlling for the classification task (object vs. action recognition) while keeping the training dataset constant (action recognition on Kinetics 400). In Figure 2a we observe that in the Early Visual Cortex, video models score significantly higher than the image models, while image models trained on action recognition may even capture less variance than the image models trained object recognition. The control for image models trained on action recognition decouples the effect of classification objective from temporal modeling, and shows that the latter the determining factor out of the two for RSA scoring in the early visual cortex. In later areas it can be seen that here the classification objective exerts more influence on the RSA score, as the video models do not fare much better than image models trained on action recognition, and those in general score higher than image models trained on object recognition. Out of all the later area groups, the Lateral stream shows the least significant differences. In relation to the noise ceilings, we observe that models are able to explain the largest amount of the total explainable variance in the Early Visual Cortex out of all brain regions (see Figure [3] Appendix for re-scaled plot by the UNC). In Figure 2b we notice that all three model categories exhibit an carly-to-late hierarchy, with more shallow layers correlating the highest with early brain regions, and deeper layers correlating the highest with late brain regions. The mid-network representations
6

---

Published as a conference paper at ICLR 2025
Image Models (ImageNet-1k)
Image Models (Kinetics-400)
Video Models (Kinetics-400)
Early Visual Cortex
Dorsal Stream
0.6
0.4
0.2
Spearman's tho
V1
V2
V3v
V3d
V3ab
IPS0
IPS1-3
RSC
Ventral-Occipital Stream
Lateral Stream
0.8
0.6
0.4
0.2
0.0
hv4
OFA
LOC
FFA
PPA
EBA
TOS
MT
STS
Brain Regions
(a)
V2
V3ab
IPS1-3
OFA
Image Models (ImageNet 1k)
0.20
Image Models (Kinetics-400)
Spearman's tho
Videa Models (Kinetics-400)
0.15
0.10
0.05
0
0.25
0.50
0.75
1
0.25
0.50
0.75
1
0.25
0.50
0.75
0.25
0.50
0.75
Depth
(b)
Figure 2: Video vs. Image models, controlling for the classification task: We find that temporal modeling is the important factor for representational alignment to early brain regions, while action space optimization for alignment to late brain regions, evaluating the RSA score for a total of 92 models. In 2a we choose the best scoring layer for each model and show all 17 brain regions. Stars at = and on top of brackets indicate statistical significance against zero and significance of p < 0.001). Horizontal dashed
p 0.05, lines show the upper and lower noise ceilings for each region. In 2b we show different model layer depths for regions chosen by zooming in significant differences from 2a and sampling region groups evenly, the rest reported in Appendix
pairwise differences respectively
0.01,
learned by the video models correlate better with V2 than those of image models, regardless if the latter are trained for action recognition. On the other hand for OFA, it is the representations in the classification layer of action recognition models that manage match the brain better, even if there is no temporal modeling involved. In Figure 3 we point out that the top models are MVIT, CSN, and SlowFast in V2, and CSN, TimeSformer, and VideoSwin in LOC. In V2 and for portion of the subjects, the model MVTT v2 S comes close to capturing almost all of the explainable variance.
Varying the architecture. Next, we compare models by varying the architecture design (CNNs vs. Transformers), keeping temporal modeling, classification task, and training dataset all constant (video action recognition models trained on Kinetics 400). In Figure we see that, in terms of the maximum score, Transformers and CNNs appear to be mostly equivalent, either of the two architectures taking the lead in different ROIs. Nevertheless, an interesting pattern appears when analyze the changes in correlation across the models' layers. In V2 Transformers compute high correlating representations at a very shallow depth in the network (around 0.1 of the total depth), while CNNs much later, around 0.6 of the total depth. In EBA, CNNs show much clearer
7

---

Published as a conference paper at ICLR 2025
V2
LOC
04
0.7
Vivio Medio (lnsicc--C
0.4
0.3
04
3
Stapoo grte
Figure 3: Ranking of all the Kinetics-400 action recognition models in RSA scoring order, against the baseline of image object recognition models. Top models in V2 are MVIT, CSN, and SlowFast, while in LOC they are CSN, TimeSrormer, and VideoSwin. Error bars signify variation across subjects. One early and one late region are shown (the two highest overall), the rest in Appendix C
hierarchy, as correlation gradually increases with model depth, while Transformers have relatively stable representations up until the classification layer, where the score increases more abruptly. Still in EBA, Transformers explain more variance than CNNs when comparing only very shallow layers.
V2
EBA
Transforeers
CNNs
Spearman's tho
0.6
Spearman's rho
Transformers
0.4
0.2
0.1
0.0
RSC
OFA
0
0.20
0.40
0.60
0.80
0.20
0.40
0.60
0.80
Brain Regions
Depth
(a)
(b)
Figure 4: Varying the architecture: CNNs and Transformers are overall equivalent but exhibit striking differences in their score distribution across layer depths. We show RSA scores of 27 CNNs and 14 Transformers, for the best scoring model layer (4a) and all layer depths (4b). Brain regions shown are those exhibiting the most significant differences, sampling evenly from brain region groups Exhaustive results can be found in Appendix C
Varying the training dataset. Here, we compare models by varying the training dataset (Kinetics 400 vs. Kinetics 710 and Kinetics 400 vs.
and keeping all other factors constant through only including the exact same models trained on the two different datasets each pair of dataset comparisons. In Figure 5a we first observe that using models trained and enhanced version of the same dataset such as Kinetics 710 in relation to Kinetics 400, largely has no effect brain alignment. However, when testing models trained on more different domain, such as significant advantage of Kinetics 400 against Somethhngg-Something-- somewhat similar pattern in LOC. In fact, this seems to show up both in shallow and deep layers of the models (Figure 5b). Considering that Something-Something--- is dataset that never shows faces and that FFA is the functionally selective region for faces, this result identifies that dataset bias from the models' training can indeed be transferred to model-brain alignment for an ROI that is functionally selective specifically for this bias.
extended
some interesting differences emerge. In particular, there is a
in FFA and OFA, and
Relation computational complexity. Last, we investigate how the brain correlation of action recognition models relates to their computational complexity (in FLOPs). In Figure O we report moderate but consistent negative correlation of model FLOPs to model-brain alignment especially in late ROIs of the Lateral and Dorsal Streams, with significant negative correlations observed in six ROIs, the top four of which are shown in 6b In 6a we observe that although the significance of the negative relation is not present in all ROIs, the relation itself is mostly consistent throughout the brain regions. A similar investigation for model parameters and model accuracy was performed but consistent significant correlations were found (see Figure 21 Appendix E)
8

---

Published as a conference paper at ICLR 2025
V2
0.8
Kinetics-400 (1)
Kinetics-400 (2)
(2)....
Kinetics-710
Sth-Sth-v2
02
0.6
Spearman's rho
04
Speaman sho
0.4
Kineckss (2)
FRA.
0.2
02
0.0
V2
V3ab
IPSO
MT
OFA
LOC
FFA
Brain Regions
Depth
(a)
(b)
Figure 5: Varying the training dataset: Training dataset biases transfer to alignment with function-
ally selective brain areas. We compare the same three models trained Kinetics 400 and Kinetics 710, and the same four models trained on Kinetics 400 and trained on Kinetics 400 are numbered as (1) and (2) to indicate two different sets of models com- pared with Kinetics 710 and Something-Somettingge Something-v2 respectively, and the comparisons are also separated with a gap. Sa the Lighest-scoring layers are shown, selecting a representative sample of regions (the rest in Appendix C and in 5b we zoom in on two regions to show all layer depths.
Models
ROI Conclation = 0 0,40 00,000
ROI
Conrlation 0.48, p 0,0004
0.3
V3v
0.3
V3d
Spearnnn's tho
0.25
hV4
OFA
085
LOC
0.0
FFA
ROI IPS0
PPA
Condiriio --0 20,0 0.0072
Comeistin -2 -2,36, p-0.101 0.0113
V3ab
0.25
0.15
IPSO
IPS1
0.15
RSC
0.1
0.1
TOS
0.05
100
300 350 400
MT
0.0
Model GFLOPs
GFLOPs
(a)
(b)
Figure 6: Relation to computational complexity: We show a significant negative correlation of model FLOPs alignment several high-level brain areas. We correlate the RSA scores of video models to model FLOPs, using the best RSA scoring layer for each model and report all brain regions 66a. as well as highlight those with the most significant correlations (6b).
DISCUSSION
Decoupling the role of temporal modeling and action space optimization. For alignment of video models brains watching videos, and for sample of models, brains, and videos, different factors are important at different stages of visual processing. In early areas, temporal modeling improves alignment, while optimizing for action recognition does not. The latter on its own can even result in slightly worse alignment than object recognition models, hinting to a potential degradation of low-level features during action optimization (image action recognition models are pretrained on ImageNet first). The benefit of temporal modeling for alignment with early brain areas is consistent with neuroscience literature suggesting that these areas are more sensitive to short time-scale change in the input, while late areas are more temporally invariant (Hasson et al. 2008 Groen et al.) 2022 Brands et al. 2024). There, improvement over object recognition image models only comes from optimizing for action recognition and is only reflected in the models' classification layer. This could mean (1) that the information captured in late brain areas is dominated by semantics and particularly action categories (Lahner al. 2024). Or (2) that current bate-of-the-ar video models

---

Published as a conference paper at ICLR 2025
fail to capture the more fine-graine high level video representations present in the brain, aside from action labels. Future work is needed to resolve these competing accounts. In line with Mineault et al. (202I), we find no indication that action information affects alignment more for areas involved in navigation such as RSC or motion area MT. Gareia et al. (2024) observed a different pattern of results from ours, finding improved alignment for video models mainly in MT and EBA. This could reflect methodological differences (e.g. IMRI dataset, dimensionality reduction method, brain alignment metric). Our findings are consistent with recent video reconstruction study (Lu et al.) 2024a) which shows that intermediate and late regions contribute more to the decoding of video semantics, while early areas more to the decoding of low-level structure and motion in the videos.
Comparing early and late representations of CNNs and Transformers. CNNs exhibit better hierarchy overall, as there is a clear mid-depth peak for early regions and gradual improvement as depth increases for late regions. Transformers however, achieve an impressive correlation to early regions even from one tenth of layer depth. This also holds for image CNNs and Transformers (Figure 19. Why is the top-aligned layer mid-depth in CNNs. and not earlier? It does not appear therefore, an interesting hypothesis is that this relates to the dynamic nature of the stimuli, as in image fMRI studies it was not observed (Nonaka et al. 2021). This hypothesis requires further validation on (multiple) other video fMRI datasets. Conversely, why are Transformers more aligned earlier in the network than CNNs? Raghu let al. (2021) showed that representations in layers 0-60 of a CNN are more similar to layers 0-30 of a Transformer, and CNN layers 60- 120 more similar to Transformer layers 30-140. Transformers also exhibited more stable representations across layers, while CNNs showed clear divide between early and late layers. In Tuli al. (00TT) more human-like error patterns of Transformers indicated a higher behavioral alignment- we find that alignment in high-level areas based on semantic repre- sentations is mostly equivalent across both architectures. In future work is also worth investigating hybrid models (e.g. Transformers using both convolution and self-attention), as there are indications these might achieve a balance between the two architectures (Figure [5).
to be matter of network size or depth (see Figures 20 22
Relating training biases and computational efficiency to alignment. Our comparison of brain alignment between models trained on different action datasets shows that dataset biases related to a certain functional selectivity can be transferred in brain alignment with the respective functionally selective brain area. This highlights the importance of choosing models trained on a dataset that is representative (1) of the videos humans watch in the experiment, and (2) of the hypotheses tested when measuring alignment. Our observation that computationally expensive models are less aligned to higher brain areas leads us to conclude that human-like semantics are more achievable with com- putationally efficient models. This is important for machine learning research that aims to build increasingly efficient models for increasingly complex tasks, as is an indication that more com- putational resources may not be needed to compute human-like high-level representations. Ideas in this direction include using the top-aligned models as starting points for further component ablations towards alignment and efficiency, as well as pruning models to preserve alignment.
Limitations and future work. We base our choice of alignment metric on the comparison of RSA and veRSA made in Conwell et al. (2022), but this was on static images and it could be worthwhile to make the same comparison for our video data, as well as comparison between other metrics. RSA does not examine possible gains in predictivity as result of linear feature re-weighting. Next, we benchmark video models that are part of library and not all publicly available video models, which limits the comparisons we could make; to show a controlled comparison of learning paradigms we would need multiples of the same architecture in supervised, contrastive, and self-supervised variants. Our results are based a single fMRI dataset, and not validated across multiple fMRI experimental setups. BOLD signals are also indirect measurements of brain activity, which in turn an indirect measurement of the brain's representations these are latent variables that cannot be measured. Our fMRI analyses were conducted in volume space: surface-based analysis could potentially provide better predictions of brain function (Glasser et al., 2016 Coalson et al. 2018). Importantly, fMRI lacks good temporal resolution which is crucial to investigate temporal modeling at finer detail. Future steps in that direction would be to make use of all temporal samples available in the fMRI data (3 TRs) and collection of BEG/MEG data for the same videos, to perform alignment benchmarking jointly on IMRI and EEG/MEG - our work sets the foundations for such future studies by already uncovering differences in models performing temporal modeling with fMRL. Given that motion and imagery cues in videos likely also engage non-visual brain regions, another promising direction for future work is to study representational alignment in areas outside visual cortex.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
Open-source code to reproduce this benchmarking study can found in the github reposi- tory https: /tts:///iihutee com/ http://rrgaaattttterrnrrerereeea brains along with detailed installation and usage instructions.
ACKNOWLEDGMENTS
This work is supported by an ELLIS Amsterdam Unit grant to IIAG. Cs acknowledges travel support from the European Union's Horizon 2020 research and innovation program under ELISE Grant Agreement No 951847.
REFERENCES
Emily J Allen, Ghislain St- Yve, Yihan Wu, Jesse I. Breedlove, Jacob Prince, Logan Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, et al. massive 7t fmri dataset to bridge cognitive neuroscience and artificial intelligence. Nature neuroscience, 25(1): 16-126, 2022.
Alex Andonian, Camilo Fosco, Mathew Monfort, Allen Lee, Rogerio Feris, Carl Vondrick, and Aude Oliva. We have so much common: Modeling semantic relational set abstractions in videos. In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XVIII 16, pp. 18-34. Springer, 2020.
Clemens G Bartnik, Christina Sartzetaki, Abel Puigseslloses Sanchez, Elijah Molenkamp, Steven Bommer, Nikolina Vukšić, and Iris IA Groen. Distinct representation of locomotive action affor- dances human behavior, brains and deep neural networks. bioRxiv, pp. 2024-05, 2024.
Domenic Bersch, Kshitij Dwivedi, Martina Vilas, Radoslaw M Cichy, and Gemma Roig. Net2brain: A toolbox to compare artificial vision models with human brain responses. arXiv preprint arXiv:2208.09667, 2022.
Amber Marijn Brands, Sasha Devore, Orrin Devinsky, Werner Doyle, Adeen Flinker, Daniel Fried- man, Patricia Dugan, Jonathan Winawer, and Iris Isabelle Anna Groen. Temporal dynamics of short-term neural adaptation across human visual cortex. PLOS Computational Biology, 20(5): c1012161, 2024.
Joao Carreira and Andrew Zisserman. Quo vadis, action recognition? a new model and the kinetics dataset. In proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp- 6299-6308, 2017.
Joao Carreira, Eric Noland, Chloe Hillier, and Andrew Zisserman. A short note on the kinetics-700 human action dataset. arXiv preprint (Xix:1907.00687 2019.
Radoslaw Martin Cichy, Aditya Khosla, Dimitrios Pantazis, Antonio Torralba, and Aude Oliva. Comparison of deep neural networks to spatio-temporal cortical dynamics of human visual object recognition reveals hierarchical correspondence. Scientific reports, 6(1):27755, 2016.
Radoslaw Martin Cichy, Gemma Roig, Alex Andonian, Kshitij Dwivedi, Benjamin Lahner, Alex Lascelles, Yalda Mohsenzadeh, Kandan Ramakrishnan, and Aude Oliva. The algonauts project: A platform for communication between the sciences of biological and artificial intelligence. arXiv preprint arXiv:1905.0567. 2019.
Radoslaw Martin Cichy, Kshitij Dwivedi, Benjamin Lahner, Alex Lascelles, Polina lamshchinina, Monika Graumann, Alex Andonian, NAR Murty, K Kay, Gemma Roig, et al. The algonauts project 2021 challenge: How the human brain makes sense world motion. arXiv preprint 2rXiv:2104.13714 2021.
Timothy S Coalson, David C Van Essen, and Matthew F Glasser. The impact of traditional neu- roimaging methods on the spatial localization cortical areas. Proceedings of the National Academy of Sciences, (15(27):63356-E66) 2018.
Christina Sartzetaki

---

Published as a conference paper at ICLR 2025
Colin Conwell, Jacob Prince, Kendrick N Kay, George A Alvarez, and Talia Konkle. What can 1.8 billion regressions tell us about the pressures shaping high-level visual representation in brains and machines? BioRxiv, pp. 2022-03 03, 2022.
Colin Conwell, Jacob S Prince, Kendrick N Kay, George A Alvarez, and Talia Konkle. A large-scale examination of inductive biases shaping high-level visual representation in brains and machines. Nature communications, 15(1):9383, 2024.
Jody Culham, Sheng He, Sean Dukelow, and Frans AJ Verstraten. Visual motion and the human brain: what has neuroimaging told us? Acta psychologica, 107(1-3):69 94, 2001.
Diana C Dima, Tyler M Tomita, Christopher J Honey, and Leyla Isik. Social-affective features drive human representations of observed actions. Elife, 1 11:75027, 2022.
Katharina Dobs, Leyla Isik, Dimitrios Pantazis, and Nancy Kanwisher. How face perception unfolds over time. Nature communicationss, 10(1): 1258, 2019.
Adrien Doerig, Rowan P Sommers, Katja Seeliger, Blake Richards. Jenann Ismael, Grace W Lind- say, Konrad P Kording, Talia Konkle, Marcel AJ Van Gerven, Nikolaus Kriegeskorte, et al. The neuroconnectionist research programme. Nature Reviews Neuroscience, 24(7):431-455 2023.
Kshitij Dwivedi, Michael Bonner, Radoslaw Martin Cichy, and Gemma Roig. Unveiling functions of the visual cortex using task-specific deep neural networks. PLoS compulational biology, 17(8):
e1009267, 2021.
Stéphane d' Ascoo,,, Hugo Touvron, Matthew L Leavitt, Ari S Morcos, Giulio Biroli, and Levent Sagun. Convit: Improving vision transformers with soft convolutional inductive biases. In Inter- national conference on machine learning, pp. 2286-2296. PMLR, 2021.
Michael Fickenberg, Alexandre Gramfort, Gaël Varoquaux, and Bertrand Thirion. Seeing all: Convolutional network layers map the function of the human visual system. Neurolmage, 152: 184-194, 2017.
Oscar Esteban, Christopher J Markiewicz. Ross W Blair, Craig Moodie, A Ilkay Isik, Asier Erra- muzpe, James D Kent, Mathias Goncalves, Elizabeth DuPre, Madeleine Snyder, et al. fmriprep: robust preprocessing pipeline functional mri. Nature methods, 16(1):111-116, 2019.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. Slowfast networks for video recognition. In Proceedings of the IEEE/CVE international conference on computer vision, pp. 6202-62 621161112222 2019.
Kathy Garcia, Emalie McMahon, Colin Conwell, Michael F Bonner, and Leyla Isik. Modeling dynamic social vision highlights gaps between deep learning and humans. PsyArXiv, 2024.
Alessandro T Gifford, Benjamin Lahner, Sari Saba-Sadiya, Martina G Vilas, Alex Lascelles, Aude Oliva, Kendrick Kay, Gemma Roig, and Radoslaw M Cichy. The algonauts project 2023 chal- lenge: How the human brain makes sense of natural scenes. arXiv preprint arXiv:2301.03198 2023.
Matthew F Glasser, Timothy S Coalson, Emma C Robinson, Carl D Hacker, John Harwell, Essa Yacoub, Kamil Ugurbil, Jesper Andersson, Christian F Beckmann, Mark Jenkinson, et al. A multi-modal parcellation of human cerebral cortex. Nature, 536(7615): 2016.
Raghav Goyal, Samira Ebrahimi Kahou, Vincent Michalski, Joanna Materzynska, Susanne West- phal, Heuna Kim, Valentin Haenel, Ingo Fruend, Peter Yianilos, Moritz Mueller-Freitag, et al. The" something something" video database for learning and evaluating visual common sense. In Proceedings of the IEEE international conference on computer vision, pp. 5842-5850, 2017.
Kalanit Grill-Spector and Rafael Malach. The human visual cortex. Annu. Rev. Neurosci,, 27(1): 649-677, 2004.
Iris IA Groen, Giovanni Piantoni, Stephanie Montenegro, Adeen Flinker, Sasha Devore, Orrin Devinsky, Werner Doyle, Patricia Dugan, Daniel Friedman, Nick F Ramsey, al. Temporal dy- namics of neural responses in human visual cortex. Journal of Neuroscience, 42540))756- 2022.
12

---

Published as a conference paper at ICLR 2025
Umut Güglü and Marcel AJ Van Gerven. Deep neural networks reveal gradient in the complexity of neural representations across the ventral stream. Journal of Neuroscience, 35(27):10005--00144 2015.
Uri Hasson, Eunice Yang, Ignacio Vallines, David J Heeger, and Nava Rubin. A hierarchy of tem- poral receptive windows in human cortex. Journal of neuroscience, 28(10):2539-2550 2008
Martin N Hebart, Oliver Contier, Lina Teichmann, Adam H Rockter, Charles Y Zheng, Alexis Kid- der, Anna Corriveau, Maryam Jaziri-Pashkam, and Chris Baker. Things-data, a multimodal collection of large-scale datasets for investigating object representations in human brain and be- havior. Elife, 12:e82580, 2023.
Joshua B Julian, Evelina Fedorenko, Jason Webster, and Nancy Kanwisher. An algorithmic method for functionally defining regions of interest in the ventral visual pathway. Neuroimage, 60(4): 2357-2364, 2012.
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, et al. The kinetics human action video dataset. arXiv preprint arXiv: 1et117055.0950 2017.
Seyed-Mahdi
Linda Henriksson, Kendrick Kay, and Nikolaus Kriegeskorte. Fixed versus mixed rsa: Explaining visual representations by fixed and mixed feature sets from shallow and deep computational models. Journal of Mathematical Psychology. 76:184-1991 14-1999 2017.
Marcie L King, Iris IA Groen, Adam Steel, Dwight J Kravitz, and Chris Baker. Similarity judg- ments and cortical visual responses reflect different properties of object and scene categories in naturalistic images. Neurolmage, 197:368-383 -828883822220 2019.
Talia Konkle and George Alvarez. A self-supervised domain domain-general general learning framework for human ventral stream representation. Nature communications, 13(1):491, 2022.
Dwight J Kravitz, Kadharbatcha S Saleem, Chris I Baker, and Mortimer Mishkin. A neural framework for visuospatial processing. Nature Reviews Neuroscience, 12(4):217-230, 2011.
Nikolaus Kriegeskorte and Pamela K Douglas. Cognitive computational neuroscience. Nature neu- roscience, 21(9): 148-1160, 2018.
Nikolaus Kriegeskorte, Marieke Mur, and Peter A Bandetrini. Representational similarity analysis- connecting the branches of systems neuroscience. Frontiers in systems neuroscience, 2:249, 2008
Emel Küçük, Matthew Foxwell, Daniel Kaiser, and David Pitcher. Moving and static faces, bodies, objects, and scenes are differentially represented across the three visual pathways. Journal of Cognitive Neuroscience, pp. 1-13, 2024.
Tom Dupré La Tour, Michael Eickenberg, Anwar O Nunez-Elizalde, and Jack L Gallant. Feature- space selection with banded ridge regression. Neurolmage. 264:119728, 2022.
Benjamin Lahner, Kshitij Dwivedi, Polina lamshchinina, Monika Graumann, Alex Lascelles, Gemma Roig, Alessandro Thomas Gifford, Bowen Pan, Sou Young Jin, N Apurva Ratan Murty, et al. Modeling short visual events through the bold moments video fmri dataset and metadata. Nature communications, 15( 15()):6241, 2024.
Kunchang Li, Yali Wang, Peng Gao, Guanglu Song, Yu Liu, Hongsheng Li, and Qiao. Uni- former: Unified transformer for efficient spatiotemporal representation learning. arXiv preprint arXiv:2201.04676 2022.
Kunchang Li, Yali Wang, Yinan He, Yizhuo Li, Yi Wang, Limin Wang, and Yu Qiao. Uniformerv2: Unlocking the potential of image vits for video understanding, In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 1632-1643, 2023a.
Kunchang Li, Yali Wang. Junhao Zhang, Peng Gao. Guanglu Song, Yu Liu, Hongsheng Li, and Yu Qiao. Uniformer: Unifying convolution and self-attention for visual recognition. IEEE Trans- actions Pattern Analysis and Machine Intelligence, 45 45(10):12811-12610 2023b.
13

---

Published as a conference paper at ICLR 2025
Ji Lin, Chuang Gan, and Song Han. Tsm: Temporal shift module for efficient video understanding. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 7083-7093, 2019.
Ze Liu, Jia Ning, Yue Cao. Yixuan Wei, Zheng Zhang, Stephen Lin, and Han Hu. Video swin trans- former. Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3202-3211, 2022.
Yizhuo Lu, Changde Du, Chong Wang, Xuanliu Zhu, Liuyun Jiang, and Huiguang He. Animate your thoughts: Decoupled reconstruction of dynamic natural vision from slow brain activity. arXiv preprint vrXiv:2405.03200 2024a.
Zitong Lu, Yile Wang, and Julie D Golomb. Achieving more human brain-like vision via human eeg representational alignment. ArXiv, 2024b.
Raja Marjieh, Pol Van Rijn, Ilia Sucholutsky, Theodore Sumers, Harin Lee, Thomas L. Griffiths, and Nori Jacoby. Words are all you need? language as an approximation for human similarity judgments. In The Eleventh International Conference on Learning Representations, 2023.
Emalie McMahon, Michael F Bonner, and Leyla Isik. Hierarchical organization of social action features along the lateral visual pathway. Current Biology, 33(23):5035-55447, 2023.
Juliette Millet, Charlotte Caucheteux, Yves Boubenec, Alexandre Gramfort, Ewan Dunbar, Christophe Pallier, Jean-Remi King, et al. Toward a realistic model of speech processing in the brain with self-supervised learning. Advances Neural Information Processing Systems, 35:
18428-33444 2022.
Patrick Mineault, Shahab Bakhtiari, Blake Richards, and Christopher Pack. Your head there to move you around: Goal-driven models of the primate dorsal pathway, Advances in Neural Information Processing Systems, 34:28757-28771, 2021.
Mortimer Mishkin, Leslie G Ungerleider, and Kathleen A Macko. Object vision and spatial vision: two cortical pathways. Trends in neurosciences, 6:414-417, 1983.
Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfreund, Carl Vondrick, et al. Moments in time dataset: one million videos for event understanding. IEEE transactions on pattern analysis and machine intelligence, 4222):502-508 2019.
Mathew Monfort, Bowen Pan, Kandan Ramakrishnan, Alex Andonian, Barry A McNamara, Alex Lascelles, Quanfu Fan, Dan Gutfreund, Rogério Schmidt Feris, and Aude Oliva. Multi-moments in time: Learning and interpreting models for multi-action video understanding. IEEE Transac- tions on Pattern Analysis and Machine Intelligence, 44(12):9434- 94449445444 2021.
Lukas Muttenthaler, Jonas Dippel, Lorenz Linhardt, Robert A. Vandermeulen, and Simon Kornblith. Human alignment of neural network representations. In The Eleventh International Conference on Learning Representations, 2023.
Hamed Nili, Cai Wingfield, Alexander Walther, Li Su, William Marslen- Willen, and Nikolaus Kriegeskorte. A toolbox for representational similarity analysis. PLoS computational biology, 10(4)=e 2014.
Soma Nonaka, Kei Majima, Shuntaro C Aoki, and Yukiyasu Kamitani. Brain hierarchy score:
Which deep neural networks hierarchically brain-like? IScience, 24(9), 2021.
David Pitcher and Leslie G Ungerleider. Evidence for a third visual pathway specialized for social perception. Trends in cognilive sciences, 2502):100-110, 2021.
Jacob S Prince, Ian Charest, Jan W Kurzawski, John A Pyles, Michael J Tarr, and Kendrick N Kay. Improving the accuracy of single-trial fmri response estimates using glmsingle. Elife, 1::7999 2022.
14

---

Published as a conference paper at ICLR 2025
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. Do vision transformers see like convolutional neural networks? Advances in neural information processing systems, 14:12116-12128, 2021.
Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib Majaj, Rishi Rajalingham, Elias Issa, Ko- hitij Kar, Pouya Bashivan, Jonathan Prescott-Roy, Franziska Geiger, et al. Brain-score: Which artificial neural network for object recognition is most brain-like? BioRxiv, pp. 407007, 2018.
Martin Schrimpf, Jonas Kubilius, Michael Lee, Apurva Ratan Murty, Robert Ajemian, and James J DiCarlo. Integrative benchmarking to advance neurally mechanistic models of human intelligence. Neuron, 108(3):413444 2020.
Robert Sekuler, Scott NJ Watamaniuk, and Randolph Blake. Motion perception. Steven's Handbook of Experimental Psychology, 1:121-176, 2002.
Karen Simonyan and Andrew Zisserman. Two-stream convolutional networks for action recognition in videos. Advances in neural information processing systems, 27, 2014.
Ghislain Yves and Thomas Naselaris. The weighte receptive field: an interpretable encoding model for complex feature spaces. Neurolmage, 10001882222 2018.
Ilia Sucholutsky, Lukas Muttenthaler, Adrian Weller, Andi Peng, Andreea Bobu, Been Kim, Bradley C Love, Erin Grant, Jascha Achterberg, Joshua B Tenenbaum, et al. Getting aligned on representational alignment. arXiv preprini arXiv:2310. 13018, 2023.
Du Tran, Lubomir Bourdev, Rob Fergus, Lorenzo Torresani, and Manohar Paluri. Learning spa- tiotemporal features with 3d convolutional networks. In Proceedings of the IEEE international conference on computer vision, pp. 4489-44999 2015.
Shikhar Tuli, Ishita Dasgupta, Erin Grant, and Thomas L Griffiths. Are convolutional neural net- works or transformers more like human vision? arXiv preprint arXiv:2105.07197 2021.
Alexander Walther, Hamed Nili, Naveed Ejaz, Arjen Alink, Nikolaus Kriegeskorte, and Jörn Diedrichsen. Reliability of dissimilarity measures for multi-voxel pattern analysis. Neuroimage, 137:188-200, 2016.
Aria Y Wang. Kendrick Kay, Thomas Naselaris, Michael J Tarr, and Leila Wehbe. Better models of human high-level visual cortex emerge from natural language supervision with large and diverse dataset. Nature Machine Intelligence, ((12)::415-144 2023a.
Liang Wang, Ryan EB Mruczek, Michael J Arcaro, and Sabine Kastner. Probabilistic maps of visual topography in human cortex. Cerebral cortex, 25(10):3911-3911 2015.
Limin Wang, Bingkun Huang, Zhiyu Zhao, Zhan Tong, Yinan He, Yi Wang, Yali Wang, and Yu Qiao. Videomae v2: Scaling video masked autoencoders with dual masking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 14549-14560, 2023b.
Bolei Zhou, Alex Andonian, Aude Oliva, and Antonio Torralba. Temporal relational reasoning in videos. In Proceedings of the European conference computer vision (ECCV), pp. 803-818, 2018a.
Jingyang Zhou, Noah C Benson, Kendrick N Kay, and Jonathan Winawer. Compressive temporal summation in human visual cortex. Journal of Neuroscience, 38(3):691-70, 2018b.
Ming Zhou, Zhengxin Gong, Yuxuan Dai, Yushan Wen, Youyi Liu, and Zonglei Zhen. large-scale fmri dataset for human action recognition. Scientific Data, 10(1):415, 2023.
15

---

Published as a conference paper at ICLR 2025
APPENDIX
A BRAIN DATASET DETAILS
Below, we provide a summary of the Methods section of the BMD dataset (Lahner et al. 2024).
Participants Ten healthy volunteers (6 female, mean age SD 27.01 + 3.96 years, sex self- reported) with normal or corrected-to-normal vision participated in the experiment. All participants gave informed consent and were screened for MRI safety. The experiment was conducted in accor- dance with the Declaration of Helsinki and approved by the local ethics committee.
Stimuli The stimulus set consisted of 1102 videos in total and were sampled from the Memento 1 10k 1ataset41. Each video was square-cropped and resized to 268x268 pixels. Videos had duration of 3 s and frame rates ranging from 15 to 30 frames per second. Videos were manually selected from the Memento 10k dataset by two human observers no static content), were filmed in a natural context, and represented a wide selection of possible events a human might witness. The 1102 videos selected for the main experiment were split into training and a testing set; 102 videos were randomly chosen for the testing set. Subjects additionally viewed separate set of colored, naturalistic videos (18 length, composed of 3-second videos) corresponding to one of five categories (faces, bodies, scenes, objects, and scrambled objects) in order to functionally localize each subject's 's category selective regions of interest (ROIs, see below).
encompass videos that contained movement (i.e.,
MRI data acquisition The MRI data were acquired with 3 T Trio Siemens scanner using 32- channel head coil. During the experimental runs, functional T weighted gradient-echo echo- planar images (EPI) were collected (TR = 1750 ms, TE= = 30 ms, flip angle 711. FOV read = 190 mm, FOV phase = 100%, bandwidth 2268 Hz/Px, resolution = 2.5 2.5 x 2.5 mm, slice gap = 10%, slices = 54, multi- acceleration factor 2, ascending interleaved acquisition). Addition- ally, a structural T1 Teiehttt image (TR = 1900 ms, TE = 2.52 ms, flip angle = 9", FOV read = 256 mm. FOV phase = 100%. bandwidth 170 Hz/px, resolution = 1.0 x 1.0 x 1.0 mm, slices 176 sagittal slices, multi-slice mode = single shot, ascending) and T2-weighted image (TR = =7970 ms, TE = 120 ms, flip angle 90p, FOV read 256 mm, FOV phase 100%, bandwidth = 362 Hz/Px, resolution = 1.0 x 1.0 x 1.1 mm, slice gap 10%, slices = 128, multi-slice mode = interleaved, scending) were obtained as high-resolution anatomical references. Dual echo fieldmaps (TR 636 ms. TE1 = 5.72 ms. TE2- = 8.18 ms, flip angle = 60%, FOV read = 190 mm, FOV phase = 100% bandwidth = 260 Hz/Px, resolution = 2.5 x 2.5 2.5 mm, slice gap = 10%, slices = 54, ascending interleaved acquisition) were acquired at the beginning of every session to post-hoc correct for spa- tial distortion of functional scans induced by magnetic field inhomogeneities. Subjects completed a total of 5 separate fMRI sessions on separate days. Session 1 consisted of structural scans, func- tional localizer runs, and functional resting state scans all interspersed. Sessions 2-5 consisted of the main functional experimental runs where the subjects viewed the training and testing set videos.
MRI data preprocessing Raw MRI data was converted to BIDS format63 and preprocessed us- ing the standardized fMRIPrep preprocessing pipeline (Esteban et al. 2019). Lahner et al. (2024) provide preprocessed data from two different pipelines, with MRI activations in Version A being calculated using Finite Impulse Response estimates, while Version B uses GI MiMsingle (Prince al., 2022]. We use Version B, as the authors recommend it for being higher quality. The full FMRIPrep generated preprocessing report is found in Lahner et al. (2024). Briefly, it consisted of anatomi- cal data preprocessing (intensity non-uniformity correction, skull-stripping, surface segmentation, and volume-based spatial registration to standard LiN1152N-in2009AAAym space) and functional data preprocessing (B0-nonuniformity correction, head-motte motion estimation, sice-time correction, and registration to the anatomical reference, with all spatial transformations concatenated into a single step), generating preprocessed BOLD runs in standard MNI space. Functional localizer scans were spatially smoothed (9 mm FWHM) while the main experimental data remained unsmoothed.
Brain response estimation A General Linear Model (GLM) was used to estimate single-trial beta estimates for each video. First, the main experimental runs were temporally interpolated from their acquisition TR of 1.75 seconds to a TR of 1 second to time-lock volume sampling to stimulus presentations. The interpolated fMRI time series, stimulus onsets, and stimulus durations (modeled with 3s durations) for each session separately were input to GLMsingle (Prince et al. 2022), which estimates tiill-triil beta values by (1) fitting an optimal Hemodynamic Response Function (HRF) to each voxel from library of HRFs, (2) identifying nuisance regressors from a noise pool that
16

---

Published as a conference paper at ICLR 2025
maximally explain variance, and (3) implementing fractional ridge regression to improve estimates in a rapid event-related design. The resulting single-trin trial beta estimates were normalized within each scanning session using the session's 's training set mean and standard deviation.
ROI definitions set of 23 ROIs (each separated by left and right hemispheres) previously known to be driven by dynamic stimuli spanning visual and parietal cortices were defined by Lahnner et al. (2024) by creating a non-overlapping parcellation composed of parcels resampled from sev- eral anatomical atlases (Glasser et al. 2016 Wang al., 2015 Julian al., 2012), into standard MNTI52NLin2009cA
space. Then, brain activations estimated from the independent functional localizer runs were used to identify the top 50% of activated voxels within each of the correspond- ing parcels. This ROI definition method facilitates inter-subject modeling by ensuring all ROIs were defined for each subject and each ROI contained the same number of voxels across subjects. The defined ROIs were VIv, Vid, V2v, V2d, V3v, V3d, hV4, V3ab, IPSO, IPS1-3, BA2, 7AL,, PFt, PFop, and MT, EBA, LOC, PPA, RSC, STS, OFA, FFA, and STS, separately for each hemisphere.
17

---

Published as a conference paper at ICLR 2025
B MODEL DETAILS
In this section we explicitly list all the models benchmarked. In the main paper only the model families are reported, whereas here all the models in those families are shown, including different backbones, sizes, and configurations.
Table 2: Exhaustive account of all models benchmarked.
Image Object Recognition
Action Recognition
CNNS
Transformers
CNNs
Transformers
AlexNet
CAiT_S
IR_CSN_L RR.CSN_R15Z R152
MVîTv2.S.
DenseNet161
CAiT XXS
IR CSN R152 BNfrozen IG65M
MViTv2 B
DenseNet201
ViT
BNIM
TimeSformer_Divv
ENiiieetNeiB3
Con
IR CSN R152 IG6SM
TimeSformer JoinIST
EfficientNetB6
DEiT_S
PP.CSN.R15.JG666
Uniformer.S
RegNetX l6gl
DEiT_B
IP_CSN_R152
Uniformer.B
RegNet Y 8gl
MViTv2 S
I3D R50
Uniformery2 B
ResNet34
MVITv2.B
BD HDD_R50_ootppo
Uniformerv2..B.k71000
ResNe150)
Swin_T
13D_R50 Re50_emb
VideoMAF_B
ResNet101
Swin S
I3D R50 gauss
VideoMAEy2 $
ResNet152
Swin
B3D 1e__R50hhoo _e0_y
VideoMAEv2.S
ResNeXt50)
Twins_pepver
R2PID_RS
VideoSwin_T
ResNeXt101
VIT S
SlowFast R50
VideoSwin S
VGG11
VIT
SlowFast.R100
VidcoSwin.B
VGG11BN
Slow_R50
VGG19
Slow RI01
VGG19BN
Slow_R50 INO_
Wee50esNet50
Slow_R50 R50_INIK, Boo_INNeeemmaauss
WideResNNet00
TaNet_
Inception
TPN_R50
Inception
TSM 6
RepVGGa2
TSM R50 dotprod
RepVGGbb2
TSM_R50_emm R50_embgauss
SeResNet50
TSM R550_550_mmm gauss
Se15sNeXt50
Tob-MebOne_s4
Xception41
X3D S
Xception71
X3D_M
@2D_R50_nopor
Spaeeell TSN TSN.Swin
C2D R101 inooollooo
C2D_R50_pool
C2D R50 6500ppol116
TSN_R50
TSN R101
TSN D161
SSN_MobOnnest
"Aailabiliity also Kinetics 710 Carreira et al.
2019
Availability also on Samething-Somethigng (Goyal et al.
2017
18

---

Published as a conference paper at ICLR 2025
C ALL REGIONS OF INTEREST
In this section we exhibit all Regions of Interest (ROIs) for analyses where subset of the ROIs was shown in the main results.
V2
V3v
V3d
0.25
Image Models (ImageNet-11
Image Models (Kinetics 400)
Spearman's tho
Videa Models (Kinetics 40%
0.15
0.10
0.05
0.25
0.50
0.75
1
0.25
0.50
0.75
0.25
0.50
0.75
0.25
0.50
0.75
Depth
N/4
OFA
LOC
FFA
PPA
Image UmmegnNet-11 Image Modils (Kiwiira-t03)
Video Model (Cinttcs 420)
Spearmnn'
0.2
0.1
Depth
V3ab
IP50
IPS1-3
RSC
0.20
Image Models (ImagaNett 1k)
Spearman's tho
Image Models (Kinetics-400)
Models (Kinetics 400)
0.10
0.05
0.25
0.50
0.75
1
0.25
0.50
0.75
0.25
0.50
0.75
0.25
0.50
0.75
Depth
EBA
TOS
MT
STS
Image Models (ImageNet-1k 1k)
0.25
Image Models (Kinetics-100)
Spearman's tho
Videa Models (Kinatics-400 400)
0.15
0.05
0
0.25
0.50
0.75
1
0.25
0.50
0.75
1
0.25
0.50
0.75
0.25
0.50
0.75
Depth
Figure 7: Expanding Figure 2b to show, for all ROIS, the comparison across layers of object recog-
nition image models and action recognition image and video models.
19

---

Published as a conference paper at ICLR 2025
Figure 8: Expanding Figure to show model ranking for all ROIs.
20

---

Published as a conference paper at ICLR 2025
CNNS
Hrarstormers
0.4
Spearman's tho
0.0
V1
V2
Vev
V3d
V3ab
IPSD
IPS1-3
RSC
0.8
0.6
0.0
hV4
OFA
LOC
FFA
PPA
EBA
TOS
MT
STS
Brain Regions
Figure 9: Expanding Figure 4a to show all ROIs for the comparison of CNNs and Transformers.
VI
V2
VBv
Vad
CNNS Transformers
Speaman's tho
0.2
0.20
0.00
0.20
0.00
Depth
CHWW: Turafomer
0..
335
545
3.6
3.30
625
0.00
0.40
0.90
0.5
0.25
3.0
8.00
000
0.50
0.90
0.00
0.20
0.60
2.00
6.5
Depth
VSab
IPS0
IPS2-3
RSC
0.20
ChiNs Transformers
Spearaa's
0.15
0.10
0.05
0.20
0.50
0.20
0.40
0.80
0.20
0.60
0.80
0.0
0.40
0.50
0.00
Depth
EBA
TOS
MT
STS
CNNS
Spearman's
Depth
Figure 10: Expanding Figure 4b to show all ROIs for the comparison of CNNs and Transformers.
21

---

Published as a conference paper at ICLR 2025
Kinelis- 400 (3)
Kineicc-400 (2)
0.6
Kinelcs-110
5th-Sth-22
0.5
0.1
Spearman's tho
0.0
V1
V2
V3v
V3d
V3eb
IPSO
IPS1-3
RSC
0.8
0.4
0.2
0.0
hv4
OFA
LOC
FFA
PPA
EBA
TOS
STS
Brain Regions
Figure 11: Expanding Figure 5a to show ROIs for the comparison of training datasets.
V3v
V3d
Spearnans tho
Sth Sthw2
0.1
0.20
0.40
0.00
0.80
0.20
0.40
0.50
0.80
0.20
0.40
0.50
0.80
0.20
0.40
0.00
0.80
Depth
DFA
LOC
FRA
PPA
Krenica-0 1itss-000022
Depth
VSab
IPS0
IPS2-3
RSC
0.25
Speamaan' tho
0.20
Sih-Stt-22
0.05
0.20
0.40
0.60
0.80
0.20
0.40
0.80
0.80
1.
0.20
0.40
0.60
0.80
0.20
0.40
0.50
0.80
Depth
Kineics-400 (2)
Spearnen's rho
Sth-Sttw22
0.20
0.40
0.80
0.80
0.20
0.40
0.60
0.80
0.20
0.70
0.60
0.80
6
0.20
0.40
0.50
0.08
Depth
Figure 12: Expanding Figure 5b
show all ROIs for the layer-wise comparison of training datasets.
22

---

Published as a conference paper at ICLR 2025
D ADDITIONAL VIEWS
In this section we include alternative ways to produce or view the results of our benchmarking, including different visualization, dimensionality reduction, or of models.
Image (ImageNet-1k 1k)
Image Models (Kinetics 400)
(Kinetics 400)
Early Visual Cortex
Dorsal Stream
0.6
Normalized Spearman's tho
0.0
V2
V3v
V3d
V3ab
IPSO
IPS1-3
RSC
Oectrall Stream
Lateral Stream
0.4
0.2
hV4
OFA
LOC
FFA
PPA
EBA
TOS
MT
STS
Brain Regions
Figure 13: Version of Figure 2a re-scaled to the Upper Noise Ceiling. With this visualization the amount of variance explained is more evident, but the information of ROI noise and absolute correlation is lost.
V2
Va
v2d
Inage Mocela (eragsher:3))
Sp mortts tho
Inage Mocels (Chneks 403)
Video Madel Jinsica-tot
Depth
MM
LOC
FFA
bru seoni- -. (lastco400)
020
5.0
6/0
V2db
P50
IP51-3
RSC
Image Mocale (Irapehann33
Speermertts rhp
Image Models (Kinets 401)
video Medel Kontess-00 400
0.20
0.40
Depth
(Iaagehee1)) Models
Speremenss Fho
Medel (onstes-400
0.2
0.
0.0
0.43
0.9
0.00
0.20
0.40
0.50
0.80
0.20
0.40
0.90
0.80
0.20
0.40
0.50
0.80
Depth
Figure 14: Version of Figure 7 with all individual models displayed separate lines.
23

---

Published as a conference paper at ICLR 2025
Speamens tto
6.1
0.30
0.40
0.0
0.20
0.40
0.50
0.80
0.20
0.40
9.50
9.80
0.23
380
Depth
OFA
Ghiss Trunstmmrss
Spearnen's tho
0.
0.2
0:85
0.00 0.00
Depth
Figure 15: Comparing hybrid Transformers that include convolution. (a) shows image object recog- nition models, and black marks ViT (d Ascoli al. (021). (b) shows video action recognition models, and black marks Uniformer (Li et al.] (202b)). Con ViT's alignment is similar to the other image Transformers, while Uniformer seems fall between video CNNs and Transformers.
sexdls
Early Visual Cortex
Dorsal Stream
Spearman's rho
0.0
V1
V2
V3v
V3d
V3ab
IPSD
IPS1-3
RSC
Ventral-Oocipital Stream
Lateral Stream
0.8
0.2
hvM
OFA
LOC
FFA
PPA
EBA
TOS
STS
Brain Regions
Image are imagy trandh Omagrionnd
1w/erta--0 s40 Wiwerrs-ooo PCA
image Miiels 90wile--000-
0.2
0.1
1.0
0.20
0.40
0.50
0.20
0.40
0.50
0.00
Depth
(b)
Figure 16: Dimensionality reduction method comparison the analysis from Figure 2 using Prin- cipal Component Analysis (PCA) vs. using Sparse Random Projection (SRP) vs. no dimensionality reduction. Overall. models' RDMs match the brain less when constructed with full dimensions than when keeping important components same with SRP where the dimensionality is still high.
24

---

Published as a conference paper at ICLR 2025
Image Models ((mageNet-1K)
Image Models (Kinetcs-400)
Early Visual
0.3
0.2
Spearnnn' tho
V1
V2
V3v
V3d
V3ab
IPSO
IP51-3
RSC
Ventral-Occipital Stream
Lateral Stream
0.6
0.2
Brain Regions
(a)
Models (Inagehen..))
0.08
Spearter's
0.85
0.02
0.20
3.00
3.60
3.80
0.25
0.40
8.70
0.00
0.20
0.40
0.30
3.30
3.20
3.00
3.00
0.00
Depth
(b)
Figure 17: Image model results (analysis from Figure 2 with the 3-repetition 1000-video training set instead of the 10-repetition 102-video test set. Both noise ceilings and model scores drop signif- icantly from Figure due increased measurement noise from the fewer stimulus repetitions.
videoman_ B
videoman B pretrainONLy
Early Visual Cortex
Dorsal Stream
Speamnnns tho
IP51-3
Ventral-Occipital Stream
Lateral Stream
0.0
hV4
OFA
LOC
FFA
PPA
EBA
TOS
MT
STS
Brain Regions
Figure 18: Comparison of the self-supervised model videomae B, with and without (video- mae B pretrainONLY) supervised finetuning on Kineties 400. Here, seems that supervised fine- tuning is crucial for alignment- however, general conclusion should not be drawn from this one model comparison and future work needed investigate this further.
25

---

Published as a conference paper at ICLR 2025
VI
V2
V3v
Vdd
0.20
imge mikk CNNS
Tardomers
Spearnan's
0.10
Depth
IW6
OFA
LOC
FRA
PPA
02
030
030
020
6.30
021
Depth
Vieb
IPS0
IPS2-3
RSC
0.15
Image inkk CNNS
Trandfomerss
Spearman's
Depth
EBA
TOS
MT
STS
imge
Spearann'
0.1
0.40
Depth
Figure 19: Layer hierarchy for image object recognition CNNs and transformers, plotted as in Figure 4b Similar to the pattern displayed by video action recognition models, here transformers also exhibit alignment peak to Early Visual Cortex much earlier in network depth than CNNs.
V2v
V3d
Image Modds
Spearman's 's rho
Inage Modils (Kiwics-103) Vides Models (chutes-400)
0.2
0.1
0.20
0.50
Depth
Figure 20: Only showing CNNs, plotted in separate lines. AlexNet, marked in black, also does not exhibit an early-layer peak in Early Visual Cortex regions. The only CNNs having an early-layer peak in VI are C2D and TPN, which both have ResNet50 backbone and so particularly shallow.
26

---

Published as a conference paper at ICLR 2025
E RELATION TO PARAMETERS & ACCURACY
In this section we provide results for the relation of the RSA score to model parameters and model accuracy.
0.35
ROI
V1
0.3
V2
0.3
V3v
V3v
Spearnnn's rho
V3d
Spearman's mo
V3d
0.25
hV4
OFA
OFA
LOC
LOC
FFA
FFA
0.2
PPA
0.2
PPA
V3ab
V3ab
IPSO
IPSO
0.15
IPS1
0.15
IPS1
RSC
RSC
EBA
TOS
TOS
0.1
20
40
60
80
120
MT
72
74
80
82
84
MT
ModelParams
STS
ModelAccuracy
STS
Figure 21: Similar to Figure 6a but with model parameters and model accuracy. Here the trends are less consistent than with the model FLOPs.
Correlation 0.09, 0.4965
Correlation 0.12, 0.3380
Layer
ROI
ROI V2
40 60
80
100
120
140
20
40
100
140
ModelParams
ModelParams
Correlation = 0.12, p 0.3644
Coreelation = 0.05, 0.6767
5.0
0.7
Layer
Layer
ROI
ROI V3d
62
80 ModelParams
60
100
120
140
20
10
60
80
100
120
140
ModelParams
Figure 22: Relation of model parameters with the top-aligned layer to the Early Visual Cortex in (all) CNNs. Early-layer correlation does not seem to be function of model size.
27


Reviews:
[{"review_id": "qQiIp91aYp", "reviewer": "Reviewer_dMT3", "summary": "This paper investigates the roles of temporal modeling and action recognition optimization in aligning video models with brain activity patterns during video observation. The findings show that early visual processing regions in the brain benefit from temporal modeling, aligning with neuroscience literature that these areas are sensitive to short-term changes, whereas late areas are more semantically oriented and responsive to action categories. Comparisons between CNNs and Transformers reveal that Transformers align with early brain regions at shallower layers due to their global attention mechanism, suggesting a different spatial processing approach than CNNs, which makes sense when compared with our intuitions about the models. The study also finds that computationally efficient models align better with high-level brain regions, implying that achieving human-like abstract semantics does not require high computational resources. Limitations include reliance on indirect brain measurements, fMRI's low temporal resolution, and a limited set of available video models.", "questions": "- Why limit the analysis to visual regions only? Motion and imagery cues in videos can activate brain networks beyond the visual cortex. \n- There are several recent visual/neural decoding papers (such as MindEye, MindVis, etc.). How do they compare numerically with what is presented in this paper?", "limitations": "", "rating": 8, "confidence": 1, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- Interesting paper, great exploration, well written. \n- The study provides a nuanced understanding of how temporal modeling and action recognition optimization affect alignment with different stages of brain processing, offering valuable insights for both neuroscience and machine learning.\n- By comparing CNNs and Transformers, the paper highlights unique alignment patterns, suggesting that global attention mechanisms in Transformers may capture early brain processing more effectively than CNNs.\n- The finding that computationally efficient models align better with high-level brain areas supports the development of optimized models that achieve human-like abstract representations without excessive computational costs.\n- The research bridges machine learning and neuroscience, potentially guiding the design of video models that better align with human brain activity (particularly in applications involving temporal and semantic processing).", "weaknesses": "- One significant limitation is the narrow focus on visual regions, neglecting other brain networks that are also engaged during video watching. Motion and imagery cues in videos can activate regions beyond visual areas, including those related to emotion, memory, and multisensory integration. This raises the question of why the study did not analyze the entire brain, as doing so could potentially yield more comprehensive insights and improve alignment accuracy by leveraging richer information from other relevant brain networks.\n- The study only benchmarks models available in a specific library, limiting comparisons across diverse training paradigms (e.g., supervised, contrastive, self-supervised). This restricts the generalizability of findings and leaves open questions about whether models trained with different methods would exhibit distinct alignment patterns.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "29tz1BRDK0", "reviewer": "Reviewer_gi1E", "summary": "The paper aims to address an important question, which is the correlation between human brain activations to complex stimuli - video-watching - recorded with fMRI and extracted features from various deep learning models. The authors compared about 100 different deep learning models (image-based, video-based) for ten different subjects. They used different criteria to evaluate the correlation between brain and deep learning models' activations: 1. temporal modelling (image-based vs video-based) 2. classification tasks (image vs video classification); 3. type of architecture (CNN vs Transformer-based models) 4. training datasets 5. computational complexity of the models. The study identifies various trends, such as the correlation between models' activations and early or high-level regions from the visual cortex, as well as differences between architectures (CNN vs Transformers) and a negative global correlation with models' complexity.", "questions": "**Questions about the limitations of the study**:\n\n- The paper discusses alternatives in the literature to study brain alignment via projecting feature space into voxel space. Still, it should be compared with more recent approaches than those cited in the paper and notably very successful methods that correlated brain activations and language modelling (speech/text), such as [1,2,3]. Is there any reason why those methods are performing in NLP, but maybe are not in image-based/video-based models?\n\n**Methods/Results**:\n\n- I understand the complexity of using high-res feature dimensionality, but it would have been nice to know/see if PCA impact the prediction accuracy (line 188) and the following results. \n\n- How is the feature extraction achieved for the transformer-based architectures (line 270)? This is not clear at the moment.\n\n- Line 275: \"Image models trained on action recognition expect the input all at once according to their preprocessing function similar to video models.\" The temporal modelling for the image-based models on video datasets is not clear. Could you please clarify?\n\n- Could you confirm that the points drawn in Figure 2.a and similar correspond to the maximum activation across layers for any given model? Would it be interesting to study the variance of correlation across layers for each model? Are there any models where layers are completely decorrelated to brain activations or is the correlation generally homogenous across layers? I am not sure this information is currently provided in the current graphs. \n\n- From the graphs in Appendix B, notably Figure 7, it seems that the layers the most responsive to V1/V2 are the mid-layer forms of most of the architectures; what is the rationale, as we know that low-level visual features are extracted by early layers in CNNs based architectures?\n\n- Given that many results indicate that the highest correlation is for the classification layer, would it be interesting to use an unsupervised algorithm? how would you expect the results to differ? \n\n- From Figure 7, it seems that the early visual cortices v1, v2, and v3 have very similar patterns, although one would expect hierarchical modelling in these particular layers. Is this the case? Is there any rational behind? \n\n**Data**\n\n- A thorough description of the dataset used is clearly missing. In particular, what is the voxel resolution, the MRI acquisition parameters, the TR of fMRI?\n\n- Is the brain solely anatomically aligned or functionally aligned? One would expect the brains to be functionally aligned, e.g. using MSM registration [4], as the study compares brain functional activation between different subjects, and anatomical registration might not be sufficient to compare brain activations at the voxel level. \n\n- Are the brains aligned across repetition of the same stimuli? It is not clear if this occurs during the same session or in different sessions. As voxels are averaged across subjects, this seems important. \n\n- Information is missing about whether the study takes into account the temporal lag between brain activation and stimuli to compare brain activation and features. Also, the limitations of using BOLD signals to model brain processing should be mentioned. \n\n- Are the ROIs defined subject-wise or using a template? This should greatly impact the correspondence of functional regions between subjects.\n \n- There is no discussion on the limit of using voxel-wise modelling compared to surface-based modelling. However, surface-based modelling has been shown to be a better predictor of brain function [5,6].\n\n\n[1] Toward a realistic model of speech processing in the brain with self-supervised learning, J.Millet et al, Neurips 2022\n\n[2] Natural speech reveals the semantic maps that tile human cerebral cortex, AG Huth et al, Nature 2016\n\n[3] Semantic reconstruction of continuous language from non-invasive brain recordings, J Tang et al, Nature Neuroscience 2023\n\n[4] Multimodal surface matching with higher-order smoothness constraints, EC Robinson et al, Neuroimage 2016\n\n[5] A multi-modal parcellation of human cerebral cortex, MF Glasser et al, Nature 2016\n\n[6] The impact of traditional neuroimaging methods on the spatial localization of cortical areas, T Coalson et al, PNAS 2018", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "- The paper discusses a very interesting and relevant topic. Such a large-scale benchmark, which compares very different and high-performing CNNs and Transformer architectures, is of high value for investigating how image and video models compare with brain feature extraction and hierarchical modelling of visual information.  \n\n- The methodology is relatively well-thought-out, with many differentiating criteria to evaluate the impact of important properties of computer vision models, which should process visual information differently: image-based and video-based modelling, different training datasets, transformers vs. CNNs, etc. There are all relevant, and the paper does a good job at analysing the results. \n\n- Statistical tests are carefully run to validate the relevance of the results, which is a great plus to support the validity of the results for such a large-scale study. \n\n- The discussion underlines important points about the role of different layers (in terms of depth), and about the difference in modelling visual features with self-attention (transformer) and convolutions (CNNs)", "weaknesses": "- The abstract and introduction lack clarity and could benefit from better writing. For instance, the concepts of \"brain representational alignment\" and \"model-brain alignment\" in the abstract are unclear and can be confusing. The second bullet point int= the contributions is not very clear. \n\n- Although the question addressed is highly important/interesting, the reason for conducting such research is not sufficiently well-motivated. Particularly, it is difficult to clearly understand the neuroscience motivations for comparing brain activations to features extracted from deep learning models and how the study would benefit both the development of neuroscience and deep learning architectures in the future. \n\n- One of the main weaknesses is the lack of description of the data acquisition and data preprocessing, which are crucial for a paper of that scale. More extensive (even in the appendix) information about the fMRI acquisition (TR, resolutions, etc.), the preprocessing strategies, and the anatomical/functional registration strategies would be required to fully evaluate how brain signals could be reliably used for comparison. Otherwise, this would undermine the quality of the results and conclusions that can be drawn (see questions below)\n\n- Some points of the methodology could benefit from more clarity (see questions below)\n\n- Some limitations of the study could be better discussed: choice of the fMRI dataset, and the limitations of using the Brain Representational Alignment to model brain alignment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "dv8fja6gsV", "reviewer": "Reviewer_Ggym", "summary": "This paper investigates representation alignment between the human visual system and task-driven deep learning models in viewing dynamic natural stimuli, measured in representational similarity across brain regions and model layers. The authors conducted a large-scale ablation on a range of vision models and training conditions, including static vs dynamic models, training data composition, and model architectures and sizes. They showed that dynamic models are more aligned with human early visual areas, moreover, convolution-based models exhibit the hierarchical structure of the biological visual system more so than Transformer-based models.", "questions": "- Figure 2a shows that image models trained on object classification are (slightly) more aligned to the early and intermediate (V3ab and hV4) visual areas than image models that are trained on action recognition, but are worse in later visual areas. I wonder if the authors have any suggestions as to why this is the case.\n- The brackets and asterisk above the box plots in Figure 2a need some spacing between pairs of boxplots. Right now they are all on the same line and it is difficult to read which pair is which. \n- The authors reported a negative correlation between models’ FLOPs (computational complexity) and alignment to higher visual areas. Does the same trend hold if we compare the model performance, maybe the validation or test accuracy (or whatever metric that was used) of the model in their respective datasets? e.g. Do more performant models in their specific task lead to worsened alignment?\n- The results show that CNNs exhibit a better hierarchy overall, while Transformers are more aligned to early visual areas. I know that ConViT is already part of the model zoo but have the authors examined architectures that use both convolution and self-attention? Such as CvT [2].\n\n[2] Wu, Haiping, et al. \"Cvt: Introducing convolutions to vision transformers.\" Proceedings of the IEEE/CVF international conference on computer vision. 2021.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The question of how similar the biological visual system and the machine learning counterpart process visual information, especially in dynamic settings, is of high interest to both the machine learning and computational neuroscience communities.\n- Overall the paper is well written. In particular, the Related Work section succinctly defines where/how this work complements existing literature. \n- The choice of models and experiments conducted are thorough, and the analysis of representation similarities between model layers and brain regions is more detailed than previous in this area. The findings on CNN-based models appear to exhibit the hierarchical structure while Transformer-based models have similar representation across depth, aligning with previous work in comparing CNN vs ViT internal representation [1]. These results can potentially guide the visual response modeling community on the choice of model architecture for different regions in the visual system.\n\n[1] Raghu, Maithra, et al. \"Do vision transformers see like convolutional neural networks?.\" Advances in neural information processing systems 34 (2021): 12116-12128.", "weaknesses": "I don’t have any major concerns with the paper but have some minor questions and suggestions, please see below.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "GXkPtg6urc", "reviewer": "Reviewer_YhDR", "summary": "This paper presents a comprehensive benchmarking study that compares deep video models with human brain responses to natural video stimuli. The authors utilize a video-fMRI dataset to investigate how various factors, such as temporal modeling, classification tasks, architecture, and training datasets, influence the alignment between neural network representations and human visual processing. The study reveals some insights into the mechanisms of video processing in both artificial and biological systems.", "questions": "The diversity of video stimuli data is lacking. Although the entire video database contains 1102 samples, the authors only used 102 for analysis. Why not use all of them? Since each video clip is only 3 seconds long and scene transitions are very limited, using only 102 samples makes it difficult to draw universally applicable conclusions.\n\nHow exactly do these comparisons motivate the design of more efficient video processing models? Based on the experimental results, it would be meaningful if the authors could give some reasonable suggestions for designing more efficient video models.\n\nIs there any project code (such as a GitHub link) to ensure the reproducibility of the experiments?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The paper addresses a novel and important question regarding the alignment of neural network representations with human brain activity in the context of video processing. The approach of systematically benchmarking a large number of models is innovative and contributes to the understanding of how different architectures and training datasets affect representational alignment.\n\nThe paper is clearly written and well-organized, making complex concepts accessible to readers. The figures and tables effectively illustrate the findings, aiding in the communication of key results.", "weaknesses": "The temporal resolution of the fMRI dataset. As mentioned by the authors, the low temporal resolution of fMRI is insufficient for representing the brain's processing of temporal information. It is recommended to consider using brain imaging data with higher temporal resolution, such as MEG, for this comparative study.\n\nThe diversity of video stimuli data is lacking. Although the entire video database contains 1102 samples, the authors only used 102 for analysis. Why not use all of them? Since each video clip is only 3 seconds long and scene transitions are very limited, using only 102 samples makes it difficult to draw universally applicable conclusions.\n\nAs mentioned by the authors, similar studies comparing brain responses to model representations on the natural scenes have been published in previous work. Although this paper tests a richer set of models, the novelty of its method and conclusions are relatively limited.\n\nThe authors mention that this work can inspire us to design more efficient video processing models. How exactly do these comparisons motivate the design of more efficient video processing models? Based on the experimental results, it would be meaningful if the authors could give some reasonable suggestions for designing more efficient video models.\n\nIs there any project code (such as a GitHub link) to ensure the reproducibility of the experiments?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)