# Emotion Recognition from Videos Using Multimodal Large Language Models
## 1 Introduction  
Thanks to its ever-increasing diffusion, video content is gradually replacing traditional textual and image web sources. Video-sharing platforms like YouTube, TikTok, and Twitch have attracted millions of social users [1]. Every day, they process millions of videos, thus requiring automated solutions for efficient and effective content classification, annotation, and retrieval. Recognizing human emotions in videos is particularly relevant to content-sharing platforms as it enables smart applications and services such as healthcare monitoring [2], AI chatbots [3], and engagement and gaming [4]. This work studies the problem of emotional reaction intensity (ERI) estimation from video sources in which the simple processing of facial expressions is not sufficient to detect the correct emotional reaction, e.g., adoration, amusement, anxiety, disgust, empathic pain, fear, and surprise. Rather than proposing ad hoc image processing techniques for emotion recognition, our purpose is to explore the capabilities of state-of-the-art multimodal learning systems that effectively combine visual and acoustic sources. To this end, this study analyzes a video benchmark collection released by the organizers of the MuSe-Reaction challenge [5]. Videos in the collection show people's reactions captured by a front-facing camera. Human reactions can be detected and evaluated by processing both a subject's face and voice. State-of-the-art approaches rely on transformer architectures [6] that jointly process the visual and audio streams. They adopt either modality-specific [7; 8] or cross-modal [9] fusion techniques to combine the separate inputs and then perform vision-language classification on top of the encoded inputs. In parallel, the progress of Large Language Models has allowed the evolution of traditional text-only Large Language Models (LLMs) towards the combined processing ofmultiple modalities. Recent LLMs such as GPT-4 [10], LLaVA [11], Video-LLaVa [12], and LaViLa [13] support visual content as part of the LLM prompts or responses beyond plain text. However, their performance on emotion recognition tasks is still largely unexplored. This paper studies the application of Multimodal Large Language Models (MLLMs) to estimate the emotional reactions in videos. It explores three alternative strategies. The first one directly applies recently proposed video-language LLMs (Video-LLaVa [12]) to estimate emotional reactions from video clips. We cast the problem of ERI estimation to a multi-regression task in which the MLLM predicts the corresponding level of intensity of each emotion. The second strategy applies probing [14] on top of MLLM embeddings. Finally, the third strategy integrates MLLM features into ViPER [9], a state-of-the-art multimodal architecture for video emotion recognition. The results achieved on the Hume-Reaction benchmark [5] give interesting insights into the performance comparison between MLLMs and traditional transformers. Despite their promising performance, MLLMs in a zero-shot setting are still incapable of achieving a higher correlation score on the analyzed video clips. The combination of MLLMs with transformer-based architectures turns out to be marginally beneficial in the average performance metrics. However, a deeper analysis of the per-class results highlights the higher capability of MLLM-based approaches to correctly estimate emotional reaction intensities that deviate from the average of the entire collection. Importantly, their higher effectiveness in predicting atypical emotion intensities could be particularly helpful in situations in which fine-tuning ad hoc models is unfeasible due to a lack of training data or limited computational resources. The remainder of this paper is organized as follows. Section 2 overviews the existing video-language LLMs and transformer-based architectures for ERI estimation from videos. Section 3 introduces the task and the benchmark dataset. Section 4 describes the LLM-based methods.