Modeling dynamic social vision highlights gaps between deep learning and humans

Authors: Kathy Garcia, Emalie McMahon, Colin Conwell, Michael Bonner, Leyla Isik
Primary area: applications to neuroscience & cognitive science
Keywords: NeuroAI, vision, fMRI, deep learning, social perception

Abstract:
Deep learning models trained on computer vision tasks are widely considered the most successful models of human vision to date. The majority of work that supports this idea evaluates how accurately these models predict behavior and brain responses to static images of objects and scenes. Real-world vision, however, is highly dynamic, and far less work has evaluated deep learning models on human responses to moving stimuli, especially those that involve more complicated, higher-order phenomena like social interactions. Here, we extend a dataset of natural videos depicting complex multi-agent interactions by collecting human-annotated sentence captions for each video, and we benchmark 350+ image, video, and language models on behavior and neural responses to the videos. As in prior work, we find that many vision models reach the noise ceiling in predicting visual scene features and responses along the ventral visual stream (often considered the primary neural substrate of object and scene recognition). In contrast, vision models poorly predict human action and social interaction ratings and neural responses in the lateral stream (a neural pathway theorized to specialize in dynamic, social vision), though video models show a striking advantage in predicting mid-level lateral stream regions. Language models (given human sentence captions of the videos) predict action and social ratings better than image and video models, but perform poorly at predicting neural responses in the lateral stream. Together, these results identify a major gap in AI's ability to match human social vision and provide insights to guide future model development for dynamic, natural contexts.

Decision: accept (poster)
Reviewer scores: 6, 8, 8, 6

Reviews and discussion:
<thread>
creation date:
1730021348994

thread_type:
official_review

<thread_object>
creator: reviewer_trmu

<content>
summary:
Unlike previous neuroscience AI studies that focus on deep learning models' responses to static images, this paper examines models' responses to human social interactions in dynamic videos. The authors create a small benchmark and evaluate an impressive number of models (over 350 image, video, and language models) on this dataset. These comprehensive experimental results could offer valuable insights for researchers in this field.

strengths:
- The relationship between AI models and the human brain is an important area of study.
- This paper presents a highly comprehensive benchmark for examining how AI models—spanning image, video, and language models—respond to social interactions compared to human brain responses in similar scenarios.
- The limitations and discussions offer valuable insights that can inform and guide future research developments.

weaknesses:
- The technical contribution is limited. While I understand the overall effort and workload invested in this paper, it does not address the question of how to develop a human-like AI model.
- Questions regarding the evaluation method: Why is a linear mapping applied between the extracted features and human data, such as fMRI responses? The rationale or assumptions behind this choice of linear mapping are not clear to me. Furthermore, is it appropriate to apply the same linear mapping approach across all models, despite their significant differences? If the evaluation method is not well-justified, the overall efforts should be reassessed carefully.

questions:
- Please include additional details about the experiments.
- It seems to remain unclear whether developing a human-aligned model would enhance existing AI models, why not test some social behavior tasks used in the computer vision community, such as the social interaction tasks in [Ego4D [CVPR2022]](https://arxiv.org/abs/2110.07058) ? (not required to conduct such experiments)

Moreover, given my limited expertise in this specific topic, I recommend that the significance of this paper be assessed by reviewers with specialized knowledge in this area.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730591911141

thread_type:
official_review

<thread_object>
creator: reviewer_xjaq

<content>
summary:
1. The authors benchmark hundreds of image, video and language models for behavior ratings and neural responses based on human social videos and their captions.
2. The authors compare these models based on their predictions of the behavior rating and neural responses.
3. The authors present several conclusion about the compared models and highlight gaps in their alignment.

strengths:
1. The authors extend a human social video dataset with human annotated text descriptions.
2. The authors benchmark 350 image, video and language models for behavior rating and neural response prediction.
3. The authors highlight gaps in alignment of these models and compare their performance along different axes like architecture, training objective etc.

weaknesses:
1. Just mentioning broad non-exhaustive categories like "self/category supervision, multi-modal and convolution vs transformer" is not a systematic approach to model selection. The authors should first select dimensions of model categorisation they want to compare like supervision type, generative or discriminative, model size, modality etc. and then choose models representing each type. The entire process needs to be detailed to make sure there is no bias in model selection that might affect conclusions downstream. The authors should provide their detailed model selection method.

2. While the authors benchmark several vision models, the vast majority of them are trained for image classification. Models trained with other objectives like object detection (only 2 used as far as I can tell), segmentation (none used as far as i can tell), masked reconstruction (none used as far as i can tell) etc. should also be benchmarked. More generally, vision based models should be categorised based on the different objectives used to train them and compared to provide insights into how different training objectives affect performance/alignment.

3.  While there are several generative language models benchmarked, the vision language models are primarily discriminative. Generative vision models like diffusion based (like stable diffusion), GAN based (like VQ-GAN) and VAE (like VQ-VAE) based models should also be benchmarked for insights into the performance gap between generative and discriminative representations for both language and vision models.

4. The comparisons between the benchmarked models need to be fair in terms of the number of parameters. It looks like the authors have compared the models regardless of the size. The authors should investigate and report whether there exists trends between prediction performance and model size for insights into how model size affects representation quality/alignment.

questions:
see weaknesses
</content>
</thread_object>

</thread>
<thread>
creation date:
1730673705002

thread_type:
official_review

<thread_object>
creator: reviewer_jqvp

<content>
summary:
The primary contribution of this work is an analysis of 350 models over an existing dataset of social actions. The labels range from user ratings to fMRI images for brain regions engaged in the watching of videos of social action. The authors find that networks trained over images outperform language/video modalities.

strengths:
1) Action datasets contain both physical and social actions, and are not focused on the exclusive modeling of either action. Exploring models trained over social actions exclusively provides very valuable insights on the difficulty of this domain of action.
2) The evaluation is very broad - 350 models is very impressive. The claims in the discussion are well supported.

weaknesses:
1) The size of the dataset is very small (200 training videos), and the results are definitely impacted by this. Models trained over video datasets in particular must be large due to the variation over the time dimension. The models (and modalities) that perform well might largely be because of the size of the dataset.
2) Audio as a modality is missing, but I would argue is just as valuable as the sequence of images alone across many of the subjects (e.g. valence, arousal). Audio provides less benefit in action datasets focused on physical actions, but might be just as important as the visual modality w.r.t. social actions.
3) There is a lack of discussion around pre-training of the different models. This bears more importance than the model architecture (CNNs vs Transformers) especially due to the small size of the dataset. The video models may or may not be trained over datasets that include social actions (like Kinetics).

questions:
1) I am unfamiliar with the usage of fMRIs in machine learning. But I imagine the variance from individual to individual must cause difficulties in predicting brain responses. Does it not make more sense to condition fMRI prediction on the baseline fMRI readings before the video viewing? Is the text or video input enough?
2) Figure 7 and Figure 8 mention image-language models being evaluated - but to my understanding, all models only take one modality. Are there models that take both images and language?
3) Action datasets deal with label subjectivity - is there any disagreement across annotators concerning the dataset being trained over? I imagine this is particularly a problem in the domain of social action.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730871422688

thread_type:
official_review

<thread_object>
creator: reviewer_rdsd

<content>
summary:
In this paper, the author extends a dataset of natural videos describing human action interactions by providing human-annotated sentences for each video and investigates the limitations of over 350+ models to predict human behavioral ratings and neural responses to dynamic social scenes. It concludes that language models predict action and social ratings better than image and video models but perform poorly at predicting neural responses in the lateral stream and provides insights into how well current AI systems replicate human social vision. More importantly, the author highlights the gap in current models' ability to understand dynamic social interactions and suggests potential directions.

strengths:
•	The paper’s approach is innovative in building on the NeuroAI benchmarking with dynamic visual responses rather than using static scene responses, which are commonly evaluated by current image model. It’s the first investigation of benchmarking many models in response to naturalistic videos of human actions.

•	The paper gives a comprehensive model evaluation experiments conducting with this dataset. Spanning from video, language, image models over 350+, including a variety of architectures and objectives.

•	The paper is fully public with all data, code, model accessible. Further, it provides an interesting direction that Human-aligned DNNs may be a promising direction for dynamic social perception, and suggests that developing models that can handle relational and temporal elements essential for social scene understanding.

weaknesses:
•	Limited Coverage or advanced video and language models: Although the dataset has been tested in 350+ models. The majority of them are obsoleted and cannot fully present the overall performance on the state-of-the-art image, video, and language models, like MViT, Co-DETR, DINO, GPT4o, LLAVA, Llama, etc. The most recent model on the paper’s list is up to 2021.

•	The author conducts experiments on a dataset, consisting of 250 3-sec videos, which is relatively small and less representative for training and evaluating deep learning models for making a significant claim on social action recognition. The data limitation might reduce the generalizability of getting conclusions when trading this complex multi-agent interactions task. 

•	While the paper make a claim that language models are successful in predicting behavioral response but not in neural responses since providing language captions is insufficient for achieving neural alignment, it encourages the society to make a better connection between neural responses and models with a more well-designed approach. Instead, the conduct of experiments are insufficient in pinpointing precise architectural or training factors that contribute to performance differences by given task.

questions:
Questions are asked in the weakness section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1731608397103

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
General review response

comment:
We are grateful to the reviewers for their thoughtful and constructive feedback on our manuscript. We have updated our manuscript with additional analyses and edits we made based on your feedback. 

We are still running some analyses to extract embeddings for additional models, including models trained on different image tasks and larger language models, and plan to upload another version with these updates next week. In the meantime, we are happy to hear your feedback on these revisions and if you have additional suggestions for how we can further improve the paper together. Thank you!
</content>
</thread_object>

</thread>


Metareview:
metareview:
This paper investigates the ability of deep learning models to predict human behavioral ratings and neural responses, proposing a dataset of human action-interactions to build on existing simpler image-based works. The dataset includes human annotated captions, allowing for use of language models as well. Over a large number of models, spanning image, video, and language models, a number of findings are reached, including the inability of vision models to predict human action and social interactions and neuronal responses along the ventral visual stream, the improvements in predicting mid-level lateral stream activity by video models, and improved ability of language models to predict action and social ratings. Overall, the proposed benchmark and large-scale experimentation seeks to improve the understanding of how current models do and do not match representations in the brain. 

  The reviewers all appreciated the idea of benchmarking NeuroAI tasks using more complex, dynamic datasets and responses rather than static scenes, as well as the comprehensive nature of the code. However, a number of shared concerns were raised including limited coverage of recent models (after 2021) as well as lack of a coherent framing/categorization for their selection, limitations on the amount/richness of data in the dataset, and lack of other modalities such as audio (which are important in human processing of real-world data). As part of the rebuttal, the authors incorporated a number of additional models including Taskonomy models and GPT models by OpenAI/Cohere (via APIs), and diffusion models. Additionally, text including justification of model selection and other details were added. Overall, reviewers were satisfied that many of their concerns were addressed.

  Considering all of the materials, I recommend acceptance of this paper. While there are still some limitations, the current version would be a benefit to the community in terms of providing a richer benchmark for this task and analysis across a range of models. Namely, the addition of more modern GPT and diffusion models significantly added to the impact of the paper. I highly encourage the authors to include the model selection justification and additional results in the camera-ready version.