Modeling dynamic social vision highlights gaps between deep learning and humans
Authors: Kathy Garcia
Emalie McMahon
Colin Conwell
Michael Bonner
Leyla Isik
Venue: ICLR 2025 Poster (2025)
Keywords: NeuroAI
vision
fMRI
deep learning
social perception
Primary area: applications to neuroscience & cognitive science
TL;DR: This study benchmarks 350+ AI models against human behavioral and neural responses to videos of social actions and highlights significant gaps in AI's ability to model dynamic social vision.

Abstract: Deep learning models trained on computer vision tasks are widely considered the most successful models of human vision to date. The majority of work that supports this idea evaluates how accurately these models predict behavior and brain responses to static images of objects and scenes. Real-world vision, however, is highly dynamic, and far less work has evaluated deep learning models on human responses to moving stimuli, especially those that involve more complicated, higher-order phenomena like social interactions. Here, we extend a dataset of natural videos depicting complex multi-agent interactions by collecting human-annotated sentence captions for each video, and we benchmark 350+ image, video, and language models on behavior and neural responses to the videos. As in prior work, we find that many vision models reach the noise ceiling in predicting visual scene features and responses along the ventral visual stream (often considered the primary neural substrate of object and scene recognition). In contrast, vision models poorly predict human action and social interaction ratings and neural responses in the lateral stream (a neural pathway theorized to specialize in dynamic, social vision), though video models show a striking advantage in predicting mid-level lateral stream regions. Language models (given human sentence captions of the videos) predict action and social ratings better than image and video models, but perform poorly at predicting neural responses in the lateral stream. Together, these results identify a major gap in AI's ability to match human social vision and provide insights to guide future model development for dynamic, natural contexts.

Published as a conference paper at ICLR 2025
MODELING DYNAMIC SOCIAL VISION HIGHLIGHTS GAPS BETWEEN DEEP LEARNING AND HUMANS
Kathy Garcia Emalie McMahon Colin Conwell', Michael F. Bonner & Leyla Isik
DDpartmentt of Cognitive Science, Department of Biomedical Engineering Johns Hopkins University Baltimore, MD 21218, USA
{kgarci18, ema 1 ememahon, cconwel12, mfbonner, lisik]e jhu. edu
"authors contributed equally
ABSTRACT
Deep learning models trained computer vision tasks widely considered the most successful models of human vision to date. The majority of work that sup- ports this idea evaluates how accurately these models predict behavior and brain responses to static images of objects and scenes. Real-world vision, however, is highly dynamic, and far less work has evaluated deep learning models on hu- man responses to moving stimuli, especially those that involve more complicated, higher-order phenomena like social interactions. Here, extend a dataset of natural videos depicting complex multi- interactions by collecting human- annotated sentence captions for each video, and we benchmark 350+ image, video, and language models on behavior and neural responses to the videos. As in prior work, we find that many vision models reach the noise ceiling in predicting visual scene features and responses along the ventral visual stream (often considered the primary neural substrate of object and scene recognition). In contrast, vision models poorly predict human action and social interaction ratings and neural re- sponses in the lateral stream (a neural pathway theorized to specialize in dynamic, social vision), though video models show striking advantage in predicting mid- level lateral stream regions. Language models (given human sentence captions of the videos) predict action and social ratings better than image and video models, but perform poorly at predicting neural responses in the lateral stream. Together, these results identify a major gap AI's ability to match human social vision and provide insights to guide future model development for dynamic, natural contexts.
INTRODUCTION
Over the past decade, significant advances have been made understanding the computations un- derlying both biological and artificial vision, in large part due to deep learning models that now provide the best match to human visual behavior and neural responses. However, most research has focused exclusively on static scene and object recognition, neglecting the rich, dynamic interactions that characterize real-world vision. Human vision is tuned to process dynamic social scenes from only few months of age (Hamlin al. 2007). yet social vision remains substantial open chal- lenge in artificial intelligence (Al) (Bolotta & Dumas, 2022), where current AI models struggle to match even human infants in their ability understand social scenes (Gandhi et al. 2022) Shu al.)
2021). Many have argued that incorporating insights from cognitive (neuro Jscience may improve Al models' performance on social tasks (Zhou et al.) 2019 McMahon & Isik 2023 Malik Isik) 2023). However, AI vision and language models rapidly evolving, and current models have never been comprehensively tested against humans in dynamic, social vision.
The human brain processes dynamic social scenes in regions that are distinct from those involved in classical object perception (Tarhan & Konkle 2020 Wurm et al. 2017 McMahon Isik 2023
Lee Masson & Isik 2021). These regions form the recently proposed lateral visual stream (Pitcher &
Ungerleider 2021 Wurm & Caramazza 2022), specialized for dynamic social perception and dis-
tinct from the classical ventral "what" and dorsal "where" streams (Ungerleider & Míshkin 1982

---

Published as a conference paper at ICLR 2025
Goodale & Milner 1992). While Al models effectively predict the ventral stream's computations, little is known about the lateral stream's 's computations. Recent work suggests computational sim- ilarities between the ventral and lateral streams, proposing that lateral stream representations may also be hierarchical, with each computational stage yielding increasingly abstract representations (McMahon et al. (023). Some work has suggested that lateral stream computations not distinct from those in the ventral stream (Finzi et al.) 2022 2023). However, this claim is based on visually-evoked responses static images, while lateral stream regions respond primarily to dynamic stimuli (Pitcher et al. 2001 Pitcher & Ungerleider 202T).
Here, we use a large-scale benchmarking approach to investigate the computational principles un- derlying human social vision and identify areas of critical need for Al model development (Figure T.. Using over 350 image, video, and language models, we predict human behavioral ratings and IMRI responses to a dataset of publicly available natural videos depicting human social actions (McMahon et al.) 2023). We find that language models based on sentence captions of the videos are best at predicting human social ratings and that video models are best, on average, predicting brain responses in the lateral visual stream. However, the performance in lateral stream regions is substantially lower than in the ventral visual stream, and model is able to match both human be- havior and brain data. Together, these results highlight a critical need for image-computale models of social perception that match human abilities.
3 videos and captions
350+ image, video and language models
Human behavior and fMRI responses
Reiing
"A girl and woman read together"
Figure 1: A summary of our overall approach. We extract representations from over 350 image, video, and language models based on three-scconds videos human social actions or their captions. We then use DNNs to predict human behavioral ratings and fMRI neural responses to the videos.
2 RELATED WORK
Our approach builds on the NeuroAl benchmarking approach that has been popularized by oth- ers (Schrimpf al., 2018 2020 Gifford al., 2023 Willeke et al. 2022 Conwell et al. [024]] Elmoznino & Bonner 2024 Chen & Bonner 2023). For cognitive neuroscience, Al bench- marking can elucidate the computational f'cctors needed to match the human brain. For AI, these benchmarks can reveal whether different algorithms are numan-aligned and suggest avenues for fu- ture model development. The aim of Ae benchmarking is often either identify the single best model of the brain or behavior (Schrimpf et al.) 2018 8 2020 Gifford et al. 2023 Willeke et al.) 2022 or to understand the computational principles underlying human-model alignment (Conwell
al. 2024 Elmoznino & Bonner 2024 Chen & Bonner 2023).
Here, we take the latter approach, but rather than aiming to understand static scene responses in the ventral visual stream, which are well modeled by most current image models (Conwell et al. 2024). we aim to understand dynamic visual responses across the brain, focusing on the lateral visual stream and human annotations of the features of interest for these regions. In addition to benchmarking vision models, we also use language models to predict behavioral ratings and visually-evoker neural responses as has been done in prior work with static scenes (Conwell al. 2023 Doerig al. 2022
2

---

Published as a conference paper at ICLR 2025
Geirhos et al. 202 Linsley al. 2023 Fel al.
2022 Lahner al. 2023 As in Conwell al.
(202)), , we use multiple language models as predictors and selectively perturb the sentence captions of the stimuli to provide insights into the kind of linguistic features that are predictive of visual responses. While modeling dynamic visual events is a growing area of interest (Lahner et al. 2023). this is the first investigation of benchmarking many models in response naturalistic videos of human actions.
3 METHODS
3.1 CODE AND DATA AVAILABILITY
All code used in this paper and our sentence captions are publicly available: /ttp::://ttt::::thtttt/s
com/ Isik- 1ab/ / S fMRI modeling. The social action ratings and fMRI responses htttts:///sf..o/////
publicly available on
with a Creative Commons Autribution 4.0 International (CC-BY-4.0) license. The videos shown participants and used here extract model activations are from the Moments in Time (MiT) dataset(http: htt////mm/etttt///mm/ s. csa 1 mm edu The MiT license restricts public release of videos from the dataset, but instructions for how obtain the videos are available in the original manuscript (McMahon et al. 2023).
3.2 SOCIAL ACTION DATASET
Here, we model behavioral ratings and neural responses from a publicly available dataset of human social actions (McMahon al.) 2023). All experimental details are included in the original publi- cation. Briefly, participants in the fMRI saw the stimuli at approximately 20 degrees of visual angle and were allowed to freely view the videos. Behavioral ratings were collected online from separate participants.
The dataset includes 250 three-second videos of social actions that are divided into 200 videos for training and 50 videos for evaluation. While this dataset is too small for training most Al models, is on par with other NeuroAT benchmarking studies Conwell et al. (2024). Each video includes human behavioral ratings of the visual and social scene features. The rated dimensions include descriptions of how large the scene is (spatial expanse), how close people in the video are (interagent distance), the extent which the people are facing another (agents facing), the extent to which an action is object-directed (object directed), the extent to which people are jointly engaged in an action (acting jointly), the extent to which they communicating (communicating)) and affective features (valence and arousal). Ratings were collected on a Likert scale by at least ten subjects, and the average rating for each feature. Inter-rater agreement is quite high and is used as the noise ceiling against which our models are evaluated.
The dataset also includes voxel-wise fMRI neural responses to each of the videos (beta values) in four participants, and an estimate of the explainable variance determined as the test-retest reliability of responses to the same videos. To restrict our analyses reliable voxels, we use the reliability mask for each subject's 's data provided with the original paper. The fMRI data also includes anatom- ically (early visual cortex. EVC, and motion-selective middle temporal area. MT) and functionally defined regions of interest (ROIs) in the ventral and lateral streams. The ventral functional ROIs include the face-selective fusiform face area (FFA) (Kanwisher et al. 1997) and place-selective parahippocampal place area (PPA) (Epstein & Kanwisher, 1998), and the lateral ROIs include the extrastriate body area (EBA), which processes bodies and relations between bodies (Downing et al.) 2001 Abassi & Papeo 2020). the lateral occipital cortex (LOC) which is object sclective (Grill- Spector et al.] 2001) and is involved in processing object-directed actions (Wurm et al., (017). 7) and posterior and anterior social-interaction selective regions in the STS (pSTS and aSTS) (Isik et al.
2017 Walbrin et al. 2018 Lee Masson & Isik] 2021 McMahon et al. 2023
3.3
SENTENCE CAPTIONING OF VIDEOS
In order to evaluate language model prediction of behavioral and neural responses, we collected sentence captions of the videos from 150 online participants using the Prolific platform and in accor- dance with the Johns Hopkins Homewood Institutional Review Board. Eligibility criteria included having completed at least 50 tasks with 85% approval rate, having normal corrected-to-nomeee
3

---

lemma with Published as a conference paper at ICLR 2025
vision, and being native English speakers. All participants were 18+ (M 39.72 years old, SD 13.24) and reported gender and race/ethnicity were as follows: 63 female, 87 male; 114 white, 14 black, 10 asian, 9 mixed race, 2 other, 3 declined to report. Participants were compensated approxi- mately $12 per hour on average for their participation.
Following informed consent, each participant captioned 12 videos presented in a random order: 10 videos from the dataset used in McMMhon et al. (2023) and 2 additional catch videos that were the same across all participants. Each video appeared with text box with grayed-out text that read "Description of the actions and interactions of the people in the video in a single sentence..." and disappeared when the subject began typing their caption. Because the study was conducted online and we did not restrict the device or browser used by participants, the visual presentation likely varied across participants.
We collected at least five unique captions for each video in the main dataset. Captions were cleaned by calculating the Pearson correlation distance between each participant and all other participants for the catch video captions in the embedding space of Hugging Face's 's fine-tuned all-MiniLM--L1- vI (Wolf et al. 2020; Wang et al. 2020). If participant's 's mean distance was outside of 2.5 standard deviations of the mean distance of all other participants for either of the two catch videos, they were removed from all subsequent analyses. M-L-M- was not reused in subsequent analyses.
3.4 DNN MODEL SELECTION
We used an "opportunistic" modeling approach from other recent NeuroAI benchmarking research Conwell et al. (024).. We selected a large set of publicly available models with a variety of modali- ties, architectures, training sets and objectives, and evaluated their performance along each of these dimensions. Image and video models were selected to represent a comprehensive cross-section of high-level visual tasks, including category supervision, self-supervision, and multimodal (image- language) training, and also include convolutional and transformer architectures. In total, we tested 348 image models from collections including Torchvision FforcWiision 2016) and Pytorch- Image- Models libraries (Wightman 2019). VISSL (Goyal et al. 2021). OpenAl CLIP (Radford et al.) 202T, and Dectectron2 (Wu et al. 2019). and 8 video models, including Facebook's SlowFast (Feichtenhofer et al., 2018) and Sformer (Bertasius et al.] (021) models. Language models included 20 prrenncetransformer and embeddings-base
models, such as GPT-2 (Radford et al.) 2019) BERT variants (Devlin et al., 2019). To additionally test the effect of large language models, we used the API from Cohere and OpenAl to extract embeddings for Cohere's 's English v3 classi- fication and clustering models, and GPT-3 large embeddings, GPT-3 small embeddings, and GPT Ada-002. For a full model list and the corresponding license information, please see the supplemen- tal files. The vision and language embeddings for multimodal models were considered separately in model evaluation (e.g., CLIP image encoder is grouped with image models and CL.IP's text encoder with language models). We tested fewer video and language models relative to image models due to their availability and computational requirements, respectively. However, this only strengthens our conclusions when either model class outperforms image models.
3.5
MODEL ALIGNMENT WITH BEHAVIORAL AND NEURAL RESPONSES
3.5.1 MODEL FEATURE EXTRACTION
We utilized DeepJuice (Conwell al. 2024 python package in alpha-release shared with us by the authors, that allows for memory- efficcien feature extraction from each layer of a DNN. We extracted the intermediate representations from every unique computational submodule (referred to here as layers) of every model, with the exception of the non-open models from Cohere and OpenAl where we could only extract a single embedding via the API. We then used GPU-optimized sparse random projection (SRP) implemented in the python package to project the activations in an approximately -ineessinnii feature space based on the Johnson-Lindensee c= 0.1 (Larsen & Nelson 2014).
All model inputs were preprocessed in a model-specific manner. For image models, we extracted activations for seven sampled frames across the three seconds of each video and then av- eraged the activations across frames from the same video. In preliminary analyses, we found that this produced almost identical results to using activations from only single frame concatenating
4

---

Published as a conference paper at ICLR 2025
activations across the seven frames. Similar to image models, for language models, extracted the activations for each caption, and then averaged the activations for the captions from the same video.
3.5.2 LINEAR MAPPING
Before fitting the linear mapping, we first 7-scored the model-SRP feature space across the samples independently for each feature in the 200-video train set defined in the original dataset (McMahon) al. 2023) and then normalized the held-out data from 50 videos by the mean and standard devia- tion from the train set. We normalized the behavioral and neural data using the same procedure.
We performed linear mapping between the normalized model-SRP feature space and the normalized behavioral or neural response using leave-one-out ridge regression optimized for the GPU as imple- mented in DeepJuice (Conwell et al. 2024). Our d-penally search space seven values sampled from a logspace of 10e to 10c". In the training set, we performed 4-fold cross-validation a full sweep of the model determine the layer that produced the highest performance on the held-out data. Performance was measured as the Pearson correlation between the predicted behavioral or neural response and the true response.
We selected the optimal model layer based on this cross-validation in the training set, and evaluated each model's performance for the optimal layer in the test set. The optimal model layer was selected separately for every behavioral rating and voxel in the brain. For the brain data, we only predicted responses in voxels that were determined have high test-retest reliability in the original dataset.
3.6 CAPTION PERTURBATION EXPERIMENT
To gain insight into what aspects of the captions produced high alignment between the language models and the brain/behavior, we performed selective perturbations to the captions as in (Conwell ct al. 2023; Kauf et al.] 2024). but instead of deletion, we performed masking to keep the overall syntactic structure of the sentence intact. We used the spaCy (Honnibal & Montani 2017) package and the en core web sm model in particular to identify the parts of speech for masking. All other mrdel--brainhehhavior mapping procedure described above.
aspects of the experiment followed
3.7 STATISTICAL ANALYSIS
To determine whether there was difference in mean performance between classes of models (e.g., image relative to language), we first computed null distribution for each model by correlating the permuted predicted response and actual response over 5,000 iterations. The same shuffling procedure used across all models. We compared the true mean difference between model classes to null distribution of mean differences. The p-value was determined by performing a two-tailed test of the true value against the permuted distributions.
The same procedure was used to determine whether a perturbation on the sentence captions signif- icantly decreased alignment between the language models and the brain/behavior, except using one-tailed test to compare true degradation (performance on the "original" minus performance on the perturbed sentence) chance.
Models that were not publicly available (e.g., Cohere variants) were not included in the statistical testing because we were not able do full sweep of these models. Note that these models do not strongly outperform other models in the brain 2 or behavior so this is unlikely to affect any conclusions from the paper.
3.8 COMPUTE SPECIFICATIONS
The ridge regression for the encoding models required a substantial amount of computational re- sources. We used an institutional high-performance computing cluster equipped with 31 A100 GPU nodes (with a mix of 40 and 80 GB memory). On average, each set of model regressions took approximately 0.38 core hours for the whole-brain results and 0.39 core hours for behavior rating results, for total of 0.77 core hours per model. To run the full suite of 365 image, video, and language models for the reported results took approximately 280 core hours. Full computational
5

---

Published as a conference paper at ICLR 2025
resources for the research project (including failed experiments and experiments not reported here) required approximately 1600 core hours.
RESULTS
4.1 BEHAVIORAL RATINGS
4.1.1 LANGUAGE MODELS CAPTURE HUMAN VISUAL SOCIAL RATINGS BETTER THAN VISION MODELS
In evaluating the models, we can compare both how the different model classes (image, video, or language) perform on average predicting each caption (Figure 2 and the top performing model for each rating (Supplemental Table T. We note that due to the larger number of image models tested, the best performing model is biased towards the image models. Despite this, we see large amount of similarity between the two metrics.
image
video
language
1.00
0.75
0.50
Score
0.25
0.00
-0.55
spatial object interagent agents communi- acting valence arousal
expanse directed distance facing cating jointly
Figure 2: Prediction performance of all models in predicting behavioral responses. Each dot is the performance of single model. The colored horizontal lines indicate the mean performance for image (pink), video (green), and language (blue) models. The horizontal gray lines the inter- subject agreement, which is approximately the maximal level that any model could be expected to perform. Brackets and asterisks indicate significantly different performance between different classes models (p < 0.05: *, < 0.01: **, p < 0.001: ***).
We find that for the visuospatial ratings (spatial expanse, interagent distance, and agents facing), model class is substantially better on average (p 0.05), but for each rating, the top performing model is an image model. In contrast, for all social and action ratings, the best performing model a language model despite the over-representation of image models in our model set. Though we note the larger, more modern models did not perform best for Table D.. For predicting ratings of object directed actions and communicating, the mean difference between language and vision models is not significant (p 0.05). For ratings of agents acting jointly and affective features (valence and arousal), language models perform better on average than image models (ps < 0.01) and video models (ps 0.001), except for valence where language models do not perform better than video models (p > 0.05).
ratings (Supplemental Figure [2]
For most ratings, video models do not perform better than image models, except for predictions of communicating, valence, and arousal (ps < 0.05). For image models, we do not see different pre- dictions based on a convolutional versus transformer architecture (Supplemental Figure S super- vised versus self-supervised learning objective (Supplemental Figure language-aligned training (Supplemental Figure 6) training dataset (Supplemental Figure 9). training objective (Supplemental Figure II). the number of tunable parameters in model (Supplemental Figure 10). Together, these results suggest that many drastically different vision models perform similarly in predicting behavioral ratings.

---

Published as a conference paper at ICLR 2025
4.1.2 HUMAN-LANGUAGE MODEL ALIGNMENT DEPENDS ON BOTH NOUN AND VERB CONTENT
To understand the features driving the relatively high performance of language models, we per- formed selective perturbations on sentence captions by removing nouns or verbs or leaving only nouns and verbs from the captions (FigureBJA We calculate the degradation in performance as the score of the original, unperturbed input captions (ra) minus the score of the perturbed input captions (rp) divided by the score of the original (ra))
A
Category Original
Linguistic perturbation
Exomple sentence
outside and talking
Shuffled
compositional structure
two and outside talking men are
No nouns
nouns
two [MASK] outside and talking
Only verbs
nouns context
[MASK] [MASK] are [MASK] [MASK] talking
No
verbs
two men [MASK] outside and [MASK]
Only nouns
verbs context
[MASK] men [MASK] [MASK] [MASK] [MASK]
shuffled
verbs
only nouns
nouns
verbs
Score degradation (o-L)
1.0
0.5
00g di 1.00
0.0
00
-0.5
spatial
object directed
Interagent ent distancee
agents facing
communi- cating
acting jointly
valence
arousal
expanse
Figure 3: A. Example sentence perturbations. B. The performance of each language model (dots) in predicting human behavioral ratings following selective perturbation of the sentence captions. The bars indicate the mean performance across models for each condition and rating. Asterisks indicate that there is a significant degradation in model-behavioral alignment following perturbation relative to the unperturbed sentence.
We find that when predicting most ratings shuffling the input captions does not decrease perfor- mance, except spatial expanse and agents acting jointly (p < 0.05), suggesting that behavior-model alignment does not rely on linguistic compositional structure (Figure 33).
We can group the remaining perturbations based on disruption to noun content (no nouns, only verbs, teals in Figure and disruptions to verb content (no verbs, only nouns, pinks in TigureB3) Prediction of most ratings is degraded by disrupting noun content (ps < 0.05). except for agents facing and communicating. Prediction of most ratings is also degraded by both verb manipulations (no verbs and only nouns, ps < 0.05), except ratings of object directed actions (p > 0.05) and interagent distance (ps 0.05). Together, these results suggest that the success of language models in predicting visual responses relies on both noun and verb content, with the exception of agents facing/communication and object-directedness, respectively.
4.2
NEURAL RESPONSES
4.2.1
VISION MODELS BEST CAPTURE NEURAL RESPONSES
As in behavior, we compare both the average performance of the models, and the best model for each ROI (Supplemental Table 2 We evaluate performance in ROIS (Figure 4 and the whole brain (Figure 5). We find that for several mid-level ROIs (MT, EBA, and LOC) and high-level ROI (pSTS), video models outperform image models on average (ps < 0.001). In both early visual cortex and aSTS, the quantitative performance gain is moderate and not significantly different (ps
7

---

Published as a conference paper at ICLR 2025
0.05), and the best performing model is image model. Within image models, did not find a notable difference in performance based on architecture (Supplemental Figure [6. training task (Supplemental Figures 17
13 21, number of tunable parameters (Supplemental Figure 20
While there is an improvement of kinetics-trained models over other models mid-level regions (Supplemental Figure T9p, kinetics-trained models all video models and the other datasets all used for training image models. Language models, including large language models (GPT-3 and Cohere, Supplemental Figure 22 performed relatively poorly overall. Together, these results further underscore the importance of video processing as critical computational factor within vision models for human alignment.
image
video
language
Score (r)
0.4
0ogs B
0.2
0.0
EVC
MT
EBA
LOC
PSTS
aSTS
FFA
PPA
Lateral ROIs
Ventral
ROIs
Figure 4: The performance of each model (dots) in predicting the average response in each ROI. The colored lines indicate the mean performance of the different classes of models, and the horizontal gray line the split-half reliability of the voxel responses each ROI averaged across participants. Brackets and asterisks indicate significantly different performance between different classes of mod- els (p < 0.05: p 0.01: #*, p < 0.001: ***).
Image models
Video models
Language models
0.4
Figure 5: Visualization of the test set encoding performance of the best performing layer in the training set for each voxel from any (A) image, (B) video, and (C) language model. This is shown on the lateral and ventral surface right hemisphere one representative participant.
Contrary to what we see in behavior, language models do not outperform vision models on average in any region (ps > 0.05), but they significantly underperform in EVC, FFA, and PPA (ps < 0.05), and there is ROI in which the best model is language model. Language perturbation experiments suggest both noun and verb content are important for prediction most ROIs (Figure 18)
Despite the relatively higher performance by video models in predicting mid-level lateral regions, we still see a striking under performance even the best models in lateral regions relative to ventral regions (Figures 4
8

---

Published as a conference paper at ICLR 2025
4.2.2 HIERARCHICAL ALIGNMENT BETWEEN MODELS AND BRAINS
Previous investigations have found a hierarchical correspondence between vision models and the brain in the ventral temporal cortex in humans
Razavi & Kriegeskorte 2014) and inferi- ortemporal cortex in macaques Yaminss et al. (014). In these studies, early model layers provide the best match to posterior brain regions and later layers better match more anterior regions. We inves- tigated whether there was a similar hierarchical correspondence between models and responses in lateral visual regions by evaluating the relative depth of each voxel in the whole brain (Supplemental
Figure| 13) and on average in our regions of interest (Supplemental Figure [4). We find that though early visual cortex is best predicted by earlier layers in the models, all other regions are predicted by layers of approximately equal depth. In contrast, whole brain results do show a hierarchy along the ventral stream, thus highlighting an additional gap between models and lateral stream regions.
DISCUSSION
We used a large set of image, video, and language models to predict human behavioral ratings and and neural responses ability to predict human responses. However, there were differences in the models that are best able predict the brain versus behavior. In particular, language models tended to be the best models of human behavioral ratings, while video models best predicted responses in lateral brain regions.
dynamic social scenes. Overall, we found notable gap in all models'
5.1
LANGUAGE AND VIDEO MODELS FOR SOCIAL SCENE UNDERSTANDING
The fact that language models align with human social ratings may suggest that humans rely on non-visual aspects of social interactions to rate social features (McMahon et al. 2023
Netanyahu
ct al. 2021). While this may be partially true, it is unlikely to be the whole answer because humans make many of these social judgments quickly and automatically (MeMahon lsik 2023 Quadflieg
& Penton- Voak 2017). Further, these behavioral ratings strongly predict visual regions of the brain (McMahon et al. 2023 McMahon & Isik (023), which the language models cannot explain. An- other reason the language models predict behavior so well may be due to the captioning prompt. By instructing participants to caption the "actions and interactions" in the videos, we may have biased the language model embeddings to better predict social action content. It is therefore somewhat sur- prising that language models do not perform even higher than reported here because, for example, nigh-communication videos are often captioned explicitly with verbs like "talking" (e.g., example caption in Figure B). and yet most language models still fall short of human agreement.
In contrast, video models provided boost to prediction in mid-level lateral regions, which is sur- prising given the lack of high- high-performing video models relative to image models in our set, but they did not predict behavioral ratings more anterior regions significantly better than image mod- els. Together, this work reveals a significant gap in even state-of-the-art models' abilities to match human social action judgments and the underlying neural substrates. These results also provide in- sights into future directions for model development that can integrate the relational structure that is readily present language with dynamic visual information.
5.2 NEUROAI IN DYNAMIC SOCIAL CONTEXTS
One major advantage of this work over most prior NeuroAl studies is the focus on dynamic, social scenes. In addition to the human- model gap. several other interesting findings come out of testing models in more ccologically valid conditions. First, prior work has suggested that affective features like valence and arousal can be extracted by relatively simple convolutional neural networks (Kragell jet al. 2019, Conwell et al., 2021). While these features may be image computable, our work shows that in dynamic events, few existing vision models match human social-affective ratings.
Further, unlike prior work with static scenes (Conwell et al. 2023 Doerig et al., 2022), language models were not able to capture brain responses in the current dataset. They perform dramatically worse than image models in predicting responses in EVC, as has been documented elsewhere (Con-l well et al. 2023), but also significantly worse in high-level ventral regions (Figures 4 5). This result calls into question strong notions of "language alignment" in visual cortex and highlights the importance of dynamic stimuli for studies of even the ventral visual stream (Haxby al. 2020).

---

Published as a conference paper at ICLR 2025
5.3 LIMITATIONS
One limitation of the current study is that the relatively small size of our dataset means it is not suitable for training AI models. Larger, more diverse datasets that can better represent the complex- ity of real-world social interactions are still needed. While some dynamic NeuroAI datasets have started to be released (Lahner et al., 2023] /Zhou et al. (2033), they do not rival the stimulus variety and dataset quality of static image A datasets (Allen et al. (022). Similarly, computer vision video datasets (Kay et al., 2017: Monfort et al. 2019; Thomee Thomee et al. 2016) also lack the size and diversity of image datasets (Deng et al. 2009 Thomee et al. (016).
Another limitation is that this work- as do most NeuroAl benchmarking studies- takes advantage of available models that differ along many factors, making it difficult to isolate the impact of any one computational factor performance. This limitation may explain why we do not see significant differences for seemingly important model factors, such as convolutional versus transformer archi- tectures (Supplemental Figures 8 [6]. We aimed to overcome this by testing wide array of models, but as noted above, our set includes relatively few video and language models compared to image models. Future work should investigate how larger language models compare human on social visual tasks, though we note here that the larger language models tested (GPT and Cohere variants) were far from the top performing model in most cases (Supplemental Tables [ more tunable parameters did not always yield higher human data predictivity (Supplemental Figures
(2 and models with
10 20
5.4 FUTURE DIRECTIONS
Moving forward, we suggest a couple avenues for advancing model development based on these results. The relative success of language and video models in matching behavior and brain respec- tively, suggest that models that explicitly represent agents, objects, and their relations over time will be critical to future modeling endeavors. While transformer models should be able to capture some of this relational information in theory, they still fall short in predicting the brain and behavior the current dataset. This failure may be due to the datasets and tasks they are trained on, which do not require the agent and object-baset based representations that humans deploy in social reasoning tasks. is possible that training more human-alignn tasks will allow these large transformer models to pick up on relevant structure in social scenes.
Hluman-aligned DNNs may be a promising direction for dynamic social perception. For example, prior work has shown that implementing neuroscience-inge -inspire circuits into transformers provides more human-aligned object tracking (Linsley et al.) 202T). Other work has focused training modern DNNs with more ecologically valid data. For example, Mineault et al. (2021) trained a 3D ResNet model on agent self-motion model dorsal stream responses. while Orhan Lake (2024)
used infant headcam data to train models on high-level visual representations. We included the latter models in our set, and while they performed relatively well in our benchmarks, they were still not able to maximally predict human social ratings or neural responses (Supplemental Tables
2
Finally, efforts to merge structured cognitive modeling (Malik & Isik| 2023 Shu et al. 2021 Ne-
tanyahu et al. (02T) with patuge-compuabble DNNs may be another fruitful direction. One example
model, a combined GNN-RNN, tracks entities and their relations over time in similar video dataset to predict human social judgments (Malik & Isik 2023). Related prior work has also suggested that generative inverse planning models may be helpful in closing the gap between human social per- ception and current DNNs (Shu et al. |202 Netanyahu et al. 2021). While most instantiations of these models are not image computable, this is an active area of model development that we believe is likely to yield advances in matching human social perception (Shi et al. 2024).
5.5 CONCLUSIONS
This work identifies human social vision as key area of need for future Al research. It also offers some promising directions for future modeling endeavors. In particular, the relative success of language and video models over image models in predicting social behavior and brain responses suggests that models combining both compositional and dynamic information may be critical for more human-like social AI.
10

---

Published as a conference paper at ICLR 2025
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our results we have made our code base publicly available on GitHub com/ Isik-lab/S SI IMRI modeling]. which includes the code and information on pre-processing steps, feature extraction, linear mapping, statistical analyses and model zoo and licenses for all models tested. The experiments are reproducible using the steps provided in the README file. The code is dependent on one additional Python library DeepJuice that is available upon request. Instructions to obtain this library are also included in the README file.
at
ETHICS STATEMENT
The research adheres to all relevant ethical guidelines. The human subjects experiments conducted for this paper were done in accordance with procedures approved by our institutional IRB. The original publicly available neuroimaging dataset was also conducted in accordance with IRB, and all human subjects were compensated for their time. The model benchmarking used open source models and all model repositories and license details are included with supplementary files.
ACKNOWLEDGMENTS
This work funded in part by NSF GREP DGB-2139757 awarded to K.G., NSF GREP DGE- 1746891 awarded to E.M., and NIMH 201MM132222 awarded to L.I.
REFERENCES
Etienne Abassi and Liuba Papeo. The Representation of Two-Body Shapes in the Human Visual Cortex. Journal of Neuroscience, 40(4):852-863, 2020. doi: 10.152//NERROSCL.1T7 19. 2019. Publisher: Society for Neuroscience.
Emily J. Allen, Ghislain St- Yee,, Yihan Wu, Jesse I.. Breedlove, Jacob S. Prince, Logan T. Dowdle, Matthias Nau, Brad Caron, Franco Pestilli, Ian Charest, Benjamin Hutchinson, Thomas Nase- laris, and Kendrick Kay. A massive 7T fMRI dataset to bridge cognitive neuroscience and arti- ficial intelligence. Nature Neuroscience, 25(1): 16-126, 16-1 January 2022. ISSN 1097-6256, 1546- 1726. doi: 10.10338/41593-022---0000 -21-00662x. URL https:/ www www.anateee... www..aattee...oatiieeeeeeeeeee 141593-021100062--
Gedas Bertasius, Heng Wang, and Lorenzo Torresani. Is Space-Time Attention All You Need for Video Understanding', June 2021. URL arXiv:2102.05095 [es].
Samuele Bolotta and Guillaume Dumas. Social Neuro Al: Social Interaction as the "Dark Mat-
ter" of AI. Frontiers in Computer Science, 4, 2022. ISSN 2624-9898. URI https:
www. froniiersin.cm wtt./orraaaalleeoooo....
20/39//fcomp.../ (022.846440)
Zirui Chen and Michael Bonner. Canonical dimensions of vision. In 2023 Conference on Cognitive Computational Newroscience, Oxford, UK, 2023. Cognitive Computational Neuroscience. doi:
10.32470/CCN..0233 1588-0. URL htt ccneuro. org/view. hwwpl/aapeee__. .amperrrppp? PaperNum=1588)
Colin Conwell, Daniel Graham, and Edward A Vessel. The perceptual primacy of feeling: Affectless machine vision models robustly predict human visual arousal, valence, and aestheties. PsyArXiv, 2021. doi: https://dii.rr// 10.3 2233//sf..o////// io/5wg4s
Colin Conwell, Jacob Prince, George A. Alvarez, and Talia Konkle. Language Models of Visual Cortex: Where do they work? And why do they work so well where they do? Journal of Vision, 23(9);5653, August 2023. ISSN 1534-7362, doi: 10. 10.11667/0..33.955653
URL https:
//vo
Colin Conwell, Jacob Prince, Kendrick N. Kay, George A. Alvarez, and Talia Konkle.
A large-scale examination of inductive biases shaping high-level visual representation in
11

---

Published as a conference paper at ICLR 2025
brains and machines. Nature Communications, 15(1):9383, October 2024. ISSN 2041- 1723. doi: 10. 14088//4447-022-5544474 Publisher: Nature Publishing Group.
httLhhts://// /ttp:::://wwww// nature. com articces/
841467-024-5314722
J. Deng, W. Dong, R. Socher, L. Li, Kai Li, and Li Fei-Fei. ImageNet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pp. 248-255, 2009. doi: (t0.0//CVR..20995200000
Jacob Devlin, Ming- Wei Chang, Kenton Lec, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, May 2019. URL lnttp://axii.
org/abs/ 111800000 04805 arXiv:18 10.04805 [cs].
Adrien Doerig, Tim C. Kietzmann, Emily Allen, Yihan Wu, Thomas Naselaris, Kendrick Kay, and
lan Charest. Semantic scene descriptions an objective of human vision, September 2022. URL
http://arxiv..rrg/ http://.s/2222..111./ 11737 arXiv:2209.11 11737 [es, q-bio].
Paul E. Downing, Yuhong Jiang. Miles Shuman, and Nancy Kanwisher. A Cortical Area Selective for Visual Processing of the Human Body, Science, 293(5339):247--2473 2001. doi: 10.1126/
science.1063414. Publisher: American Association for the Advancement of Science.
Eric Elmoznino and Michael F. Bonner. visual cortex benefit from high latent dimensionality.
High-performing neural network models of PLOS Computational Biol- doi: 10.1371/joumaal.phhi
ogy, 2001)110117 January 2024. ISSN 1553-7358.
1011792. URL https: //oonnaal www.pplsss..mmoooooooooooo.ooo.. org /p
1371/journal ptt..pppii............1......... Publisher: Public Library of Science
Russell Epstein and Nancy Kanwisher. cortical representation of the local visual environment.
Nature, 39246676):5986611, April 1998. ISSN 0028-0836, 1476-4687. doi: 10.1038/33402.
Christoph Feichtenhofer, Haoqi Fan, Jitendra Malik, and Kaiming He. SlowFast Networks for Video
Recognition. arXiv: 18/2.03982, 2018.
Thomas Fel, Ivan F Rodriguez Rodriguez, Drew Linsley, and Thomas Serre. Harmoniz- ing the object recognition strategies of deep neural networks with humans. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh (eds.), Advances in Neu- ral Information Processing Systems, volume 35, pp. 9432 9446. Curran Associates, Inc., http:///eee...i..oooo/ooeeeeee/r ffteesppppe////2//// 2022. URL http://0....4.000000000222777110
file/
Dawn Finzi, Daniel Yamins, Kendrick Kay, and Kalanit Grill-Spector. Do deep convolutional neu- ral networks accurately model representations beyond the ventral stream? In 2022 Conference Cognitive Computational Neuroscience, San Francisco, 2022. Cognitive Computational Neu- roscience. doi: 10.32470ICCN.2022..2999 URL
paper. Saperrppp225. aaa.ppppaapeNNum111.1111
Dawn Finzi, Eshed Margalit, Kendrick Kay, Daniel L. K. Yamins, and Kalanit Grill-Spector. A single computational objective drives specialization of streams in visual cortex, Decem- ber 2023. URL https:: wwww biorxiv. org
572460v1
Pages: 12.3.12..9. Section: New Results.
Kanishk Gandhi, Gala Stojnic, Brenden M. Lake, and Moira R. Dillon. Baby Intuitions Benchmark (BIB): Discerning the goals, preferences, and actions of others, February 2022. URL http:
// //axxiv.or//bb.g 2102. arXiv:2102. [es].
Robert Geirhos, Kantharaju Narayanappa, Benjamin Mitzkus, Tizian Thieringer, Mathiis Bethge, Felix A. Wichmann, and Wieland Brendel. Partial success in closing the gap between hu- and machine vision, October 2021.
arXiv:2106.07411 [cs, b-bio
A. T. Gifford, B. Lahner, S. Saba-Sadiye M. G. Vilas, A. Lascelles, A. Oliva, K. Kay, G. Roig, and R. M. Cichy. The Algonauts Project 2023 Challenge: How the Human Brain Makes Sense of Nat- ural Scenes, July 2023. URL http:/ arxiv. wwtp/bs/////ooooooot/t..ot...eet abs/ /2001, 03198. arXiv:2301.0311 [cs, b-bi..
12

---

Published as a conference paper at ICLR 2025
Melvyn A. Goodale and A. David Milner. Separate visual pathways for perception and ac- tion. Trends in Neurosciences, 15(1):20-25) 1992. ISSN 3166-2236, doi: httss//doi.org 11001600166-22366922200
URL (RLLtttp:/// http:///wwwwwwweeieeeee...... ctc.i//errecccccccccccccc//eee/e science/
Priya Goyal, Quentin Duval, Jeremy Reizenstein, Matthew Leavitt, Min Xu, Benjamin Lefaudeux, Mannat Singh, Vinicius Reis, Mathilde Caron, Piotr Bojanowski. Armand Joulin, and Ishan 2021.
Misra. Vissl. http://///.stcceec///e/ccceeeeea
Kalanit Grill-Spector; Zoe Kourtzi, and Nancy Kanwisher. The lateral occipital complex and its role in object recognition. Vision Research, 41(10-11):1409-14444 1422, May 2001. ISSN 00426989. elsevier com/
doi: 10. 10.00116500442266999000000001010 kerrierr//11//0002899900000
URL http:::://lininnne
J. Kiley Hamlin, Karen Wynn, and Paul Bloom. Social evaluation by preverbal infants. Nature, 450
(7169):557-553 November 2007. ISSN 1476-4687. doi: 10.1038/nature06688.. URL https:
//80i.0rg/10.103/// wtw.:////r......ttaaaruuuureeeee
James V. Haxby, M. Ida Gobbini, and Samuel A. Nastase Naturalistic stimuli reveal a dominant role for agentic action in visual representation. Neurolmage, 216: 216:116561, August 2020. ISSN 1095-9572. doi: 10.101 10ol///.eerriiagee.....
16561.
Matthew Honnibal and Ines Montani. spaCy 2: Natural language understanding with Bloom em- beddings, convolutional neural networks and incremental parsing. To appear, 2017.
Leyla Isik, Kami Koldewyn, David Beeler, and Nancy Kanwisher. Perceiving social interactions in the posterior superior temporal sulcus. Proceedings of the National Academy of Sciences, 114(43), October 2017. ISSN 0027-8424, 1091-6490 doi: 10. 10.1073/paa..714441111
1ermnas..11144 1714471114. URL.
org. (toei/full/100 1073/ /ttp/////0.111111111000//2111111
Nancy Kanwisher, Josh McDermott, and Marvin M. Chun. The Fusiform Face Area: A Module in Human Extrastriate Cortex Specialized for Face Perception. The Journal of Newroscience, 17(11): 4302 4311, June 1997. ISSN 0270-6474, 1529-2401. doi: 10. 10.1522/NEEURRSSII111111 1997. URL https:/ /www. www.::uuuoooooooooeeeeoos
17 11 04302 1997
Carina Kauf, Greta Tuckute, Roger Levy, Jacob Andreas, and Evelina Fedorenko. Lexical-Semantic Content, Not Syntactic Structure, Is the Main Contributor to ANN-Brain Similarity of fMRI Re- sponses the Language Network. Neurobiology Language, 4368. doi: 10. 1662/nol a a.00116 16. URL.https:///ot wtwp:/0........... 1162/ nol a_00016
5(1): 5(1):7-42 April 2024. ISSN 2641-
Will Kay, Joao Carreira, Karen Simonyan, Brian Zhang, Chloe Hillier, Sudheendra Vijaya- narasimhan, Fabio Viola, Tim Green, Trevor Back, Paul Natsev, Mustafa Suleyman, and Andrew Zisserman. The Kinetics Human Action Video Dataset. arXiv: 1705.06950, 2017.
Seyed-Mahdi Khaligh-Razavi Razavi and Nikolaus Kriegeskorte. vised, Models May Explain IT Cortical Representation. e1003915, November 2014. ISSN 1553-7358. doi: 10.
Deep Supervised, but Not Unsuper- PLOS Computational Biology, 10(11): 10b.00001......0011..11 URL
https::/dxx.plos.orr org/ 137.///urraaaa pcbi 1003915
Philip A. Kragel, Marianne C. Reddan, Kevin S. LaBar, and Tor D. Wager. Emotion schemas are embedded in the human visual system. Science Advances, 5(7))caaw4358 July 2019, doi: 1.1126//ciadv..aw44333 URL /ttttt::///// www.sceeee................ wew.eennerrrgooo//eeeeeeoo 11. / aaw4358 Publisher: American Association for the Advancement of Science.
Benjamin Lahner, Kshitij Dwivedi, Polina lamshchinina, Monika Graumann, Alex Lascelles, Gemma Roig, Alessandro Thomas Gifford, Bowen Pan, You Jin, N. Apurva Ratan Murty, Kendrick Kay, Aude Oliva, and Radoslaw Cichy. BOLD Moments: modeling short visual events through video fMRI dataset and metadata. Nature Communications, 15:6241-6266 March 2023. doi: 14.1100%/444477004455000110000 URL https://///........ httEps::/ww.... nat nature. com/ articles/ s4 -41467-024-00111
13

---

Published as a conference paper at ICLR 2025
Kasper Green Larsen and Jelani Nelson. The Johnson-Lindenstrauss lemma is optimal for linear dimensionality reduction, November 2014. URL arXiv:1411.2404 [cs, math].
Haemy Lee Masson and Leyla Isik. Functional selectivity for social interaction perception in the human superior temporal sulcus during natural viewing. Neurolmage, 245:118741, De- cember 2021. ISSN 10538119. doi: 10. 10.10066..ccummmee00011111111111
URL https:
//linkinghub.. elweviereeee..... htt.a///........rcaec.////////aa
Drew Linsley, Girik Malik, Junkyung Kim, Lakshmi N Govindarajan, Ennio Mingolla, and Thomas Seme. Tracking without reconginnn in humans and machines, 2021. BtRL/htts::////
org/abs/ 2ata./105.... 13351
Drew Linsley, Ivan F. Rodriguez Rodriguez, Thomas Fel, Michael Arcaro, Saloni Sharma, Margaret Livingstone, and Thomas Serre. Performance-optimized deep neural net- works are evolving into worse models of inferotemporal visual cortex. Advances
in Neural Information Processing Systems, 16:28873-2889) December 2023. URL neur wttp:e/rrpppcccpppprreereeeeeeee 5bf 3
Manasi Malik and Leyla Isik. Relational visual representations underlie human social inter- action recognition. Nature Communications, 14(1):7317, November 2023. ISSN 2041-
1723. doi: 10. 100009/444467-0--44333333
URL https:/ www..natu... com/ articles/
64 s414 841467-023-431555
Number: Publisher: Nature Publishing Group.
Emalie McMahon and Leyla Isik. Seeing social interactions. Trends in Cognitive Sciences, 27(12):
1165-1179, December 2023. ISSN 13646613. doi: 1i..0066.iicc..2222.00. 001. URL. https:
Emalie McMahon, Michael F. Bonner, and Leyla Isik. Hierarchical organization social action features along the lateral visual pathway. preprint, PsyArXiv, March 2023. URL https://
osf. 10.1o/// /3vbb
Patrick J Mineault, Shahab Bakhtiari, Blake Aaron Richards, and Christopher C Pack. Your head is there to move you around: Goal-driven models of the primate dorsal pathway. In A. Beygelzimer, Y. Dauphin, P. Liang, Jnd Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, 2021. URL httpt:://pppreeeeeeee
net htrr/oormmmmmmmmm -in---dd-ppp--- peet/Eorumidd-pp-u0a TJ
Mathew Monfort, Alex Andonian, Bolei Zhou, Kandan Ramakrishnan, Sarah Adel Bargal, Tom Yan, Lisa Brown, Quanfu Fan, Dan Gutfruend, Carl Vondrick, and others. Moments in Time Dataset: one million videos for event understanding. IEEE Transactions on Puttern Analysis and Machine Intelligence, pp. 1-8, 2019. ISSN 0162-8828. doi: 10. 1091 21119..9944444
Aviv Netanyahu, Tianmin Shu, Boris Katz, Andrei Barbu, and Joshua B. Tenenbaum. PHASE: PHysically-groumdd Abstract Social Events for Machine Social Perception, March 2021. URL waboo//////////oooaa/eooooeaeeee abs 2203 01 933 arXiv:2103.01923 [cs, stat].
wwwp:/wwwwooroooooooooo http:://axxv..rrt
A. Emin Orhan and Brenden M. Lake. Learning high-level visual representations from child's perspective without strong inductive biases. Nature Machine Intelligence, pp. 271-283, 2024. doi: 10. 100000000622255004000000022 URL https: www.w...au.r...mmmmmmm. wtt.:/attt.......tt.ttt....e.
s4 64226--044-0662--4411-e
David Pitcher and Leslie G. Ungerleider. Evidence for Third Visual Pathway Specialized for Social Perception. Trends in Cognitive Sciences, 25(2):100-110, February 2021. ISSN 13646613, doi: 10. 10.10116j tics. 2020. 11 .0060 URL htttps://linkiggitt elsevier. wwweeviierroorrtrreeeeeeee cww/retrievee pii/ 011//533366666 0023836646611233022707
David Pitcher, Daniel D. Dilks, Rebecca R. Saxe, Christina Triantafyllou, and Nancy Kanwisher. Differential selectivity for dynamic versus static information in face-selective cortical regions. Neurolmage, 56(4):2356 2363, June 2011. ISSN 1095-9572. doi: mouroigg.e 03.067.
14

---

Published as a conference paper at ICLR 2025
Susanne Quadflieg and Ian S. Penton- Voak. The Emerging Science of People- Watching: Forming Impressions From Phrty Encounters. Current Directions in Psychological Science, 26(4): 383-389, August 2017. ISSN 0963-7214. doi: 10.1177096372144169955555 URL https: otddi.org//...
Publisher: SAGE Publications Inc.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and llya Sutskever. Language models are unsupervised multitask learners. OpenAl Blog, 2019. URL models
https://cdn.com openai wtt.://oo///eeeeeeee are_ wtw.uusuuerviieede [ multitaskk_earrerrs. pddf
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agar- wal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and llya Sutskever. Learning Transferable Visual Models From Natural Language Supervision, February 2021.
org/
wwg/abs///ooooooooooooo.oa.o.o./ abs 003. C0020 arXiv:2103.0002) [cs].
Alexander Sax, Jeffrey O. Zhang, Bradley Emi, Amir Zamir, Silvio Savarese, Leonidas Guibas, and Jitendra Malik. Learning to Navigate Using Mid-Level Visual Priors. In Proceedings of the Con- ference on Robot Learning, pp. 791-812. PMLR, May 2020. URL htttps://prott htttps://proceeddngss ISSN: 2640-3498.
mlr
html
Martin Schrimpf, Jonas Kubilius, Ha Hong, Najib J. Majaj, Rishi Rajalingham, Elias B. Issa, Ko- hitij Kar, Pouya Bashivan, Jonathan Prescott-Roy. Franziska Geiger, Kailyn Schmidt, Daniel L. K. Yamins, and James J. DiCarlo. Srore: Which Artificial Neural Network for Ob- ject Recognition most Brain-Like? preprint, Neuroscience, September 2018. URL http:
//biorxiv.org/lenr www.://o/wloooo/ceeeooooo www.ooowwooooooooooooooooo 100.1..11114000000
Martin Schrimpf, Jonas Kubilius, Michael Lee, Apurva Ratan Murty, Robert Ajemian, and James DiCarlo. Integrative benchmarking to advance neurally mechanistic models of hu- man intelligence. Neuron, 2020. URL https: http::::/www..ell.cco hot.:///o/ooo/o/uuu////ttt/
50896-6273 (20) 30605-X
Haojun Shi, Suyu Ye, Xinyu Fang, Chuanyang Jin, Leyla Isik, Yen-Ling Kuo, and Tianmin Shu. Muma-tom: Multi-modal multi-agent theory of mind, 2024. URL htttps://arxiv.om abs/2408. 12574
Tianmin Shu, Abhishek Bhandwaldar, Chuang Gan, Kevin Smith, Shari Liu, Dan Gutfreund, Eliz- abeth Spelke. Joshua Tenenbaum, and Tomer Ullman. AGENT: A Benchmark for Core Psycho- logical Reasoning. In Proceedings of the 38th International Conference on Machine Learning, pp. 9614-9625. PMLR, July 2021. URL htttps://proceeeeee ngs. ml press/v139/ ISSN: 2640-3498.
shu21a htmll
Leyla Tarhan and Talia Konkle. Sociality and interaction envelope organize visual tion representations. Nature Communications, 11(1):3002, June 2020. ISSN 2041-1723,
doi: 10.1038/s41467-022 10000-066...66-2-0---6------- 8:1467-020-1684--
16846-w. URL
https: /ttpps://www... /www wwww.natuee. nat cwm http/articcees/
Bart Thomee, David A. Shamma, Gerald Friedland, Benjamin Elizalde, Karl Ni, Douglas Poland, Damian Borth, and Li-Jia Li. YFCC100M: The New Data in Multimedia Research. Com-
munications of the ACM, 59(2):64-73, January 2016. ISSN 0001-0782, 1557-7317.
doi:
10.1145/2812802. URL http::tttap/taa/ orq/ abs 15033.01880 arXiv:1503.01811 [es].
Viscon. Torchvision: Pytorch's computer vision library.
https://github. com/
pytorch/vision 2016.
Leslie G. Ungerleider and Mortimer Mishkin. Two cortical visual systems. In David Ingle, Melvyn A. Goodale, and Richard J. W. Mansfield (eds.), Analysis of Visual Behavior, pp. 549- 586. The MIT Press, 1982.
Jon Walbrin, Paul Downing, and Kami Koldewyn. Neural responses visually observed social interactions. Neuropsychologia, 1 112:31-39, April 2018. ISSN 1873-3514. doi: 1016f. 0.iropyychooogiaa2218..00222....
15

---

Published as a conference paper at ICLR 2025
Wenhui Wang. Furu Wei, Li Dong, Hangbo Bao, Nan Yang, and Ming Zhou. MinILM: Deep Self-
Attention Distillation for Agmostei Compression of Pre-Traines Trained Transfomers, April 2020.
URLhttp:: http:///axiiiii httpxxxxvvv..... org/ abs/abs..222 10957 arXiv:2002.10957 [es].
Ross Wightman. Pytorch image models. pytorch- saerrchh-maggeee
https://github.com// http:////oowwigggaataaa.aaaaaa
1s 2019.
Konstantin F. Willeke, Paul G. Fahey, Mohammad Bashiri, Laura Pede, Max F. Burg, Christoph Blessing, Santiago A. Cadena, Zhiwei Ding, Monstantin-Klemann Lurz, Kayla Ponder, Taliah Muhammad, Saumil S. Patel, Alexander S. Ecker, Andreas S. Tolias, and Fabian H. Sinz. The Sensorium competition on predicting large-scale mouse primary visual cortex activity, June 2022.
URL htttt://aaaxii org/ abs 2206.2222222 08666 arXiv:2206.0886 [cs, q-bio].
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma. Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. HuggingFace's ' Transformers: State- of-the-art Natural Language Processing, July 2020. URL http::ttpaaaaa /ttppi.///axvv..o// org/ abwpg//bb//91.. arXiv:1910.03771 [cs].
03771
Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan-Yen Lo, and Ross Girshick. Detectron2. 2019.
Moritz F. Wurm and Alfonso Caramazza. Two 'what' pathways for action and object recogni- tion. Trends in Cognitive Sciences, 26(2):103-116, February 2022. ISSN 13646613. doi:
10.10166j.tics.220000
URL htttp::/inkinghuu..b. www.eeee.iiiii..eeeeee.......... www.:/ier..o///eeeeee/ee.
p11/S16466131002588
Moritz F. Wurm, Alfonso Caramazza, and Angelika Lingnau. Action Categories in Lateral Oceip- itotemporal Cortex Are Organized Along Sociality and Transitivity. Journal of Neuroscience, 373))562-555, 2017. ISSN 0270-6474. doi: 10. 1522/NEUROSCI..7771166
URL Publisher: Society for Neuro-
https: /wwpw//wwwww./////////////////w/ www.rosci..og///oooooo.o... :ttg://omteeett/333//5555555
science .eprint: http:s//wwwwjjieeroocccco///oons
Daniel L. K. Yamins, Ha Hong, Charles F. Cadieu, Ethan A. Solomon, Darren Seibert, and James J. DiCarlo. Performance-optimizze hierarchical models predict neural responses in higher visual cortex. Proceedings of the National Academy of Sciences, 111(23):8619-866 June 2014. ISSN 0027-8424, 1091-6490.
doi: 10..073/pnas.44031111 1403112111 URL pettps://pnas. http:/ppaa...rro.../////
1403112111
Amir R. Zamir, Alexander Sax, William Shen, Leonidas J. Guibas, Jitendra Malik, and Sil- vio Savarese. Taskonomy: Disentangling Task Transfer Learning. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), pp. 3712-3722. CVPR, 2018. URL http://///pppacccessteeeeeeeeeee com/ content ovpr 2018/ html /aamir Taskonomy_Diaee Satpooomm_iiiieeeeo DisentaanglnngggTeaaaeeee CVPR 2018 20____2aa1___.____1 html
Chen Zhou, Ming Han, Qi Liang, Yi-Fei Hu, and Shu-Guang Kuai. A social interaction field model accurately identifies static and dynamic social groupings. Nature Human Behaviour, 3(8):847- 855, August 2019, ISSN 2397-3374. doi: 10. 1038//41162-019-6666 Publisher: Nature Publishing Group.
URL httpt::/ :tttp:::/wwww
nat nature.c wttp:/cccaaaata..oo
s41552 019 0618 2
Ming Zhou, Zhengxin Gong, Yuxuan Dai, Yushan Wen, Youyi Liu, and Zonglei Zhen. A large-scale iMRI dataset for human action recognition. Scientific Data, 10(1):415, June 2023. ISSN 2052- 4463. doi: 10. 159-03434199--0--2222233 URL http:::://// ////:///wwwww... are com/ htt//aticlle Publisher: Nature Publishing Group.
081597-023-0235--
16

---

Published as a conference paper at ICLR 2025
A SUPPLEMENTAL MATERIALS
Table I: A table of the top-30 performing models averaged across features. Models are arranged in descending order of their overall average performance. Bold scores highlight the model that is the top performing model for a given feature, and the absence of a bold score for a given feature indicates that the top performing model is not in the overall top-30 performing models.
Box rumreer
Modd same
Model clas
spelid espurse
dijet vijeddrees)
ineregent (-MARO
EgoEs fecing
eeting jairlly
veleace
langraga
01588
0.675
0.364
0.675
0.311
0.725
Toomhose Al peberts-aaaee and
langpage
01717
0.457
0.550
11840
11572
06633
0/665
0814
laggagg
0.62
astusdlend si andha-eeneedd
0.766
0,530
0.50
0.541
0.51
0.706
0.67
0.588
epe Jane prrheattegs
01709
0.655
0350
0663
DAT1
0811
11717
055
0771
0.56
10
gpt apl.adl...o annbaliing
0.681
0659
0.544
0.36
0.61
0100
0,03
0635
dedikno haee sage---bbliioddee cmellr
0.07
15
chppyetli
11771
0.60
1.400
0350
6771
0.00
muscchaad-- coceet- brgo-dt
0.500
01228
0601
0A31
059
rpt)
nitudleced slanehaeeeteeee legevv
0.500
0.558
0.533
0.465
0.461
0.30
0/31
055
17
gy32
lagginge
0.19
0.685
0.512
0.326
0.45
050
6.749
0.54
bose
leagengs
11:141
0.481
0.53
0/61
089
21
andh aet-dastimbens--ae Swae
laggaags
11568
0500
0.000
11.08
0.356
0379
0.750
0661
al-Mrel Ni-Ltv
11856
0009
0.75
11280
01402
0811
0.99
082
co
23
laggagg
0.641
0/53
0.588
0.305
0.432
0.64
0.618
0.50
othon angidh a Fullete dng
laageage
0.640
0,632
0.557
0.306
0.444
0.63
0.61
0.53
alimberta largert
11:29
0.82
0.357
0887
0.64
0.0
Laisie
0,500
0.363
11214
0.766
0.64
0623
0.66
langrage
0.663
0.371
0.340
30
at 22
langrage
0.31
0.655
0.501
0.358
0.355
0.00
0.733
0.515
Table 2: A table of the top-30 models on averaged across all ROIs. Models are arranged in descend- ing order of their overall average performance in all reliable voxels. Bold scores highlight the top score in each ROI. The bottom row indicates the performance in predicting the ROIs based on hand engineered features from (McMahon et al. 2023)
Rank Model name
Mudel class
EVC
MT
EBA
LOC
PSTS
aSTS
FFA
PPA
x3d.s
video
0.390
0.440
0.427
0.352
0.251
0.249
0.510
0,518
timm timm_heity2 2eitt/_Jese_ppii 12,
image
0.369
0.245
0.349
0.326
0.375
0.293
0.536
0.588
image
0.376
0.259
0.355
0.330
0.287
0.291
0.524
0.593
video
0.460
0.336
0.255
0.280
0.512
0,492
beit.aei www..higgeeeeeacchheeeeee.e pasch166.22
image
0.373
0.241
0.335
0.314
0.227
0.297
0.500
0,592
timm Joige hwt..//gge.uucch1.__.. _pach16__544
image
0.368
0.251
0.336
0.317
0.251
0.266
0.501
0.593
clip.vin144
image
0.376
0.223
0.325
0.289
0.245
0.278
0.526
0.574
13d r50 timm
video
0.378
0.433
0.465
0.345
0.254
0.239
0.16
0.442
huge putch14 in21fkk
imago
0.320
0.291
0.264
0.287
0.587
10
timm deit3 large pww.pp...6....221ll
image
0.376
0.241
0.328
0.292
0.253
0.276
0.517
0.574
11
timm beil_baee pwwchl_.1116111
image
0.365
0.228
0.322
0.298
258
0.267
0.522
0.57%
12
rimm Jeeit Jerirt_Jrgeeppun
image
0.371
0.241
0.337
0.295
0.745
01.74
0.499
0.584
13
150
video
0.408
0.421
0.529
0.248
0.245
0.512
0.471
timm_comeext.aa Jarge
image
0.318
0.284
0.252
0.273
0.502
0.589
15
image
0.373
0.230
0.318
0.284
0.252
0.275
0.502
0.559
16
timm commest www.eeeeetaaageeeniieeee inarge_in22k
image
0.370
0.219
0.303
0.258
0.256
0.265
0.515
0.585
17
c50
video
0.371
0.376
0.405
0.305
0.258
0.272
0.511
0.462
18
timm deit3 medium patch16 224 in2k
image
0.363
0.228
0.348
0.284
0.249
0.270
0.522
0.568
timm comvect alange in226ik
image
0.285
0.217
0.268
20
clip.m5th.4
image
0.376
0.229
0.316
0.277
0.237
0.272
0.552
0.568
21
timm jommmnst Jange Jange_in222 Jn22k
image
0.368
0.225
0.307
0.269
0.249
0.282
0.520
0.555
22
timm mniser J616, 234_miil 2i1kkk jn2kk
image
0.399
0.219
0.304
01.54
0.238
0.292
0.542
0.559
23
timm heit. hase potch16 384
image
0.358
0.230
0.331
0.292
0.235
0.256
0.512
0.568
24
timm_comeeet. hase..n222ik..
image
0.324
0.275
0.240
0.258
0.517
0.549
25
him__anneest_ao hose
image
0.364
0.225
0.324
0.275
0.240
0.258
0.517
0.549
26
timm deit3, hase
image
0.362
0.217
0.321
0.286
0.224
0.257
0.539
0.576
27
slowfast .30
video
0.363
0.435
0.432
0.313
0.242
0.250
0.474
0.461
28
clip vith32
image
0.376
0.203
0.300
0.246
0.262
0.241
0.507
0.568
timm
image
0.209
0.289
0.369
0.218
0.269
0.526
0.566
30
timm_oomeestt in22
image
0.362
0.217
0.308
0.276
0.228
0.250
0.515
0.559
McMahon 2025
hand engineered
0.347
0.365
0.336
0.361
0.292
0.418
0.435
17

---

Published as a conference paper at ICLR 2025
Image
Image-Language
1.0
0.8
0.6
0.4
Score
0.2
0.0
-02
spatial expanse
object
interagent
agents communi-
acting jointly
valence arousal
directed distance facing
cating
Figure 6: The performance models in our set trained either with an image-based (image, pink) or multimodal (image-language, blue) objective function in predicting behavior ratings. Plotting conventions are same as Figure 2
convolutional
transformer
1.0
0.8
0.6
0.4
Score
0.2
0.0
0.2
spatial expanse
object interagent agents communi-
acting
valence
arousal
directed distance facing
cating
jointly
Figure 7: The performance each image model in our set in predicting behavioral responses sepa- rated by whether the model has convolutional (orange) or transformer (green) architecture. Plotting conventions are the same as Figure 2
18

---

Published as a conference paper at ICLR 2025
supervised Seff-superiised
1.0
0.8
Score (r)
0.6
0.4
0.2
0.0
spatial expanse
object
interagent
agents
communi-
acting jointly
valence arousal
directed distance
facing
cating
Figure 8: The performance of each image model in our with a ResNet-50 backbone in predicting behavioral responses separated by whether model uses supervised (red) or self-supervised (yellow) learning objective. This analysis was restricted to single architecture class (ResNet-50) to focus specifically on the learning objective. Plotting conventions are the same as Figure 2
101000 InageRitiie
0.6
Openk- 42ON
Imagenetto:
Score (n)
Imagenetti
0.4
Image-eerk
COCO
Kmetik
-0.2
spattal
object Interagent agents communi- acting facing
valence
jointly
Figure The performance of the vision models (image and video) at predicting the behavioral responses grouped according to their training data. Plotting conventions are the same as Figure 2
19

---

Published as a conference paper at ICLR 2025
spatial expanse (r
object directed Interagent distance agents facing 0.11)
0.26)
0.20)
Score in
video language
0.0
communicating
acting jointly (r= 0.35)
valence (r 0.27)
(r = 0.19)
(r = 0.33)
Score (n)
0.5
Parameters (1e8) Parameters (1e8) Parameters (1e8) Parameters (1e8)
Figure 10: For each behavioral response, the model's prediction score is plotted against the number of trainable parameters. The r-value below the behavioral response indicates the Pearson correlation between the score and number of trainable parameters.
tasiensmy, axteenced
tadonmyy 0905 oblect
tadonmy 2005 sceee
teakenaryy coreiiu
Jenetarg
1.00
0.75
spmraion
Score (r)
0.50
talunnrry Ialtdd
Calunury rpardhg
tsdentiyy Igmw
0.25
tadiensry eppsimas
(2don1ryy eypilns33
1150mry perfatte_pooee
1edmmrr temal
Leaknnrryy meddinn
0.25
tedonatry cluin
spatial object Interagant agents communik acting valence arousal espanse
fecing cating jointly
talunarry warbtinggstte
Figure II: The performance of the taskonomy (Sax al. 2020 Zamir et al. 2018 models in predicting behavioral responses. Plotting conventions are the same Figure
20

---

Published as a conference paper at ICLR 2025
cohere english classification
gpt3 apt3_large
gol_eda_ 2
cohere anglish v3, clustering
gpt3_ apt3_small smail
paraphrase-MinilMM MiniLM-16-22 (Best Benchmark Model)
0.9
0.8
0.6
0.5
0.3
spatial expanse
object directed
interagent distance
agents facing
communi- cating
acting jointly
valence
arousal
Figure 12: The performance of the GPT-3, GPT Ada-002, and Cohere models compared to the overall best performing model across behavioral ratings. Plotting conventions are the same as Figure
Figure 13: Relative depth of the best performing model layer across all vision models (image and video models) in the whole brain of one representative subject.
21

---

Published as a conference paper at ICLR 2025
1.0
0.8
Relative layer depth
0.4
0.2
0.0
EVC
EBA
LOC
PSTS
aSTS
Lateral ROIs
Figure 14: The relative layer depth of the best performing model layer for each image and video model (thin gray lines) and the average best layer depth across models (thick purple line) in each ROI along the lateral visual stream.
Image
image-Language
0.6
Score (r)
0.4
0.3
0.1
EVC
MT
LOC
PSTS
aSTS
PPA Ventral ROls
FFA
Lateral ROIs
Figure 15: The performance of models in our set trained either with an image-based (image, pink) or multimodal (image-language, blue) objective function in predicting neural responses in ROIs in the lateral and ventral visual streams. Plotting conventions the same as Figure 4
22

---

Published as a conference paper at ICLR 2025
convolutional
transformer
Score (A)
0.4
0.2
0.0
EVC
MT
EBA
LOC
PSTS
aSTS
PPA Ventral ROls
FFA
Lateral ROIs
Figure 16: The performance each image model our set in predicting neural responses ROIs in the lateral and ventral visual streams. Models are grouped by convolutional (orange) or transformer (green) architecture, Plotting conventions are the same as Figure 4
supervised self-supervised
0.5
Score (M)
0.3
0.2
EVC
MT
EBA
LOC
PSTS
aSTS
FFA
PPA
Lateral ROIs
Ventral ROIs
Figure 17: The performance by each image models with ResNet-50 backbone in predicting neural responses in ROIs in the lateral and ventral visual streams. Models are grouped by supervised (red) self-supervised (yellow) learning objective. This analysis restricted to single architecture class (ResNet-50) to focus specifically on the training objective. Plotting conventions are the same Figure 4
23

---

Published as a conference paper at ICLR 2025
shuffled
only verbs
only nouns
no nouns
no verbs
Score degradation (4-1)
0.8
0.4
60
sede
gxdia 4000
DAP 89
op
-0.2
EVC
MT
LOC
PSTS
aSTs
FFA Ventral
Lateral ROIs
Figure 18: The average performance in ROIs of each language model (dots) in predicting neural responses following selective perturbation the sentence captions. The bars indicate the mean performance across models for each condition and rating. Asterisks indicate that there is significant degradation model-ncurgy alignment following perturbation relative to the unperturbed sentence.
0.6
OpenA4400N
0.5
Imagenetion
Score
YFCCISM
0.3
Imagehattic bn1
Imageheccc RISACIAAPL
0.2
2017
EVC
MT
EBA
LOC
PSTs
aSTS
FFA
PPA
Lateral ROIs
Ventrall ROIS
Figure 19: The performance the vision models (image and video) at predicting neural responses grouped according to their training data. Plotting conventions are the same as Figure
24

---

Published as a conference paper at ICLR 2025
-0.39)
0.09)
0.01)
0.20)
0.6
Score (n)
image video language
0.2
PSTs (r = 0.03)
aSTS (r = -0.05)
FBA (r -0.26)
PPA -0.07)
0.6
Score (r)
0.4
0.2
20 Parameters (1eB) Parameters (1e8) Parameters (1eB) Parameters (1e8)
20
20
20
Figure 20: For each ROI response, the model's prediction score is plotted against the number of trainable parameters. The r- r-value next to the ROI response indicates the Pearson correlation between the score and number of trainable parameters.
cbere
donesing
decth suckdear
deatt
0.5
udge
Scor
Ispaicing
koppo rb22
knypo
nethadngg
0.0
regment semrntic
romm Layar
segment unoucds
EVC
NT
EBA
LOC
5575
1575
FRA
PPA
resment uropodd
Lateral hois
Ventrall Ricls
vanaring paint
Figure 21: The performance of the taskonomy Sax et al. 2020 Zamir et al. 2018) models in predicting neural responses in lateral and ventral ROIs. Plotting conventions the same Figure
25

---

Published as a conference paper at ICLR 2025
cohere english v3 classification
got3 large
apl_ada 2
cohero anglish va_clustering
got3_smail
paraphrase-MinlLNe (Best Model)
0.6
0.5
Score
0.1
0.0
EVC
EBA
LOC
PSTS
aSTS
FFA
PPA
Lateral
Ventral
Figure 22: The performance of the GPT-3, GPT Ada-002, and Cohere models compared to the overall best performing model. Plotting conventions are the same Figure
26


Reviews:
[{"review_id": "mTlgnwgEUp", "reviewer": "Reviewer_RdsD", "summary": "In this paper, the author extends a dataset of natural videos describing human action interactions by providing human-annotated sentences for each video and investigates the limitations of over 350+ models to predict human behavioral ratings and neural responses to dynamic social scenes. It concludes that language models predict action and social ratings better than image and video models but perform poorly at predicting neural responses in the lateral stream and provides insights into how well current AI systems replicate human social vision. More importantly, the author highlights the gap in current models' ability to understand dynamic social interactions and suggests potential directions.", "questions": "Questions are asked in the weakness section.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "•\tThe paper’s approach is innovative in building on the NeuroAI benchmarking with dynamic visual responses rather than using static scene responses, which are commonly evaluated by current image model. It’s the first investigation of benchmarking many models in response to naturalistic videos of human actions.\n\n•\tThe paper gives a comprehensive model evaluation experiments conducting with this dataset. Spanning from video, language, image models over 350+, including a variety of architectures and objectives.\n\n•\tThe paper is fully public with all data, code, model accessible. Further, it provides an interesting direction that Human-aligned DNNs may be a promising direction for dynamic social perception, and suggests that developing models that can handle relational and temporal elements essential for social scene understanding.", "weaknesses": "•\tLimited Coverage or advanced video and language models: Although the dataset has been tested in 350+ models. The majority of them are obsoleted and cannot fully present the overall performance on the state-of-the-art image, video, and language models, like MViT, Co-DETR, DINO, GPT4o, LLAVA, Llama, etc. The most recent model on the paper’s list is up to 2021.\n\n•\tThe author conducts experiments on a dataset, consisting of 250 3-sec videos, which is relatively small and less representative for training and evaluating deep learning models for making a significant claim on social action recognition. The data limitation might reduce the generalizability of getting conclusions when trading this complex multi-agent interactions task. \n\n•\tWhile the paper make a claim that language models are successful in predicting behavioral response but not in neural responses since providing language captions is insufficient for achieving neural alignment, it encourages the society to make a better connection between neural responses and models with a more well-designed approach. Instead, the conduct of experiments are insufficient in pinpointing precise architectural or training factors that contribute to performance differences by given task.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9wTN7YAYOD", "reviewer": "Reviewer_jqVp", "summary": "The primary contribution of this work is an analysis of 350 models over an existing dataset of social actions. The labels range from user ratings to fMRI images for brain regions engaged in the watching of videos of social action. The authors find that networks trained over images outperform language/video modalities.", "questions": "1) I am unfamiliar with the usage of fMRIs in machine learning. But I imagine the variance from individual to individual must cause difficulties in predicting brain responses. Does it not make more sense to condition fMRI prediction on the baseline fMRI readings before the video viewing? Is the text or video input enough?\n2) Figure 7 and Figure 8 mention image-language models being evaluated - but to my understanding, all models only take one modality. Are there models that take both images and language?\n3) Action datasets deal with label subjectivity - is there any disagreement across annotators concerning the dataset being trained over? I imagine this is particularly a problem in the domain of social action.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) Action datasets contain both physical and social actions, and are not focused on the exclusive modeling of either action. Exploring models trained over social actions exclusively provides very valuable insights on the difficulty of this domain of action.\n2) The evaluation is very broad - 350 models is very impressive. The claims in the discussion are well supported.", "weaknesses": "1) The size of the dataset is very small (200 training videos), and the results are definitely impacted by this. Models trained over video datasets in particular must be large due to the variation over the time dimension. The models (and modalities) that perform well might largely be because of the size of the dataset.\n2) Audio as a modality is missing, but I would argue is just as valuable as the sequence of images alone across many of the subjects (e.g. valence, arousal). Audio provides less benefit in action datasets focused on physical actions, but might be just as important as the visual modality w.r.t. social actions.\n3) There is a lack of discussion around pre-training of the different models. This bears more importance than the model architecture (CNNs vs Transformers) especially due to the small size of the dataset. The video models may or may not be trained over datasets that include social actions (like Kinetics).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "onmRDXT7hY", "reviewer": "Reviewer_XJaQ", "summary": "1. The authors benchmark hundreds of image, video and language models for behavior ratings and neural responses based on human social videos and their captions.\n2. The authors compare these models based on their predictions of the behavior rating and neural responses.\n3. The authors present several conclusion about the compared models and highlight gaps in their alignment.", "questions": "see weaknesses", "limitations": "", "rating": 8, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. The authors extend a human social video dataset with human annotated text descriptions.\n2. The authors benchmark 350 image, video and language models for behavior rating and neural response prediction.\n3. The authors highlight gaps in alignment of these models and compare their performance along different axes like architecture, training objective etc.", "weaknesses": "1. Just mentioning broad non-exhaustive categories like \"self/category supervision, multi-modal and convolution vs transformer\" is not a systematic approach to model selection. The authors should first select dimensions of model categorisation they want to compare like supervision type, generative or discriminative, model size, modality etc. and then choose models representing each type. The entire process needs to be detailed to make sure there is no bias in model selection that might affect conclusions downstream. The authors should provide their detailed model selection method.\n\n2. While the authors benchmark several vision models, the vast majority of them are trained for image classification. Models trained with other objectives like object detection (only 2 used as far as I can tell), segmentation (none used as far as i can tell), masked reconstruction (none used as far as i can tell) etc. should also be benchmarked. More generally, vision based models should be categorised based on the different objectives used to train them and compared to provide insights into how different training objectives affect performance/alignment.\n\n3.  While there are several generative language models benchmarked, the vision language models are primarily discriminative. Generative vision models like diffusion based (like stable diffusion), GAN based (like VQ-GAN) and VAE (like VQ-VAE) based models should also be benchmarked for insights into the performance gap between generative and discriminative representations for both language and vision models.\n\n4. The comparisons between the benchmarked models need to be fair in terms of the number of parameters. It looks like the authors have compared the models regardless of the size. The authors should investigate and report whether there exists trends between prediction performance and model size for insights into how model size affects representation quality/alignment.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "9nEP3MKd1v", "reviewer": "Reviewer_tRMU", "summary": "Unlike previous neuroscience AI studies that focus on deep learning models' responses to static images, this paper examines models' responses to human social interactions in dynamic videos. The authors create a small benchmark and evaluate an impressive number of models (over 350 image, video, and language models) on this dataset. These comprehensive experimental results could offer valuable insights for researchers in this field.", "questions": "- Please include additional details about the experiments.\n- It seems to remain unclear whether developing a human-aligned model would enhance existing AI models, why not test some social behavior tasks used in the computer vision community, such as the social interaction tasks in [Ego4D [CVPR2022]](https://arxiv.org/abs/2110.07058) ? (not required to conduct such experiments)\n\nMoreover, given my limited expertise in this specific topic, I recommend that the significance of this paper be assessed by reviewers with specialized knowledge in this area.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- The relationship between AI models and the human brain is an important area of study.\n- This paper presents a highly comprehensive benchmark for examining how AI models—spanning image, video, and language models—respond to social interactions compared to human brain responses in similar scenarios.\n- The limitations and discussions offer valuable insights that can inform and guide future research developments.", "weaknesses": "- The technical contribution is limited. While I understand the overall effort and workload invested in this paper, it does not address the question of how to develop a human-like AI model.\n- Questions regarding the evaluation method: Why is a linear mapping applied between the extracted features and human data, such as fMRI responses? The rationale or assumptions behind this choice of linear mapping are not clear to me. Furthermore, is it appropriate to apply the same linear mapping approach across all models, despite their significant differences? If the evaluation method is not well-justified, the overall efforts should be reassessed carefully.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)