# Emotion Recognition from Videos Using Multimodal Large Language Models
## 3 Task and Dataset Description  
The task of this study is the recognition of emotional reactions in videos. For this research, the Hume-Reaction dataset, a large-scale, multimodal dataset designed explicitly for the Emotional Reactions Sub-Challenge (MuSe-Reaction) [5], was employed. The dataset is notable for its extensive collection of naturalistic emotional reactions. The dataset  
\begin{table}
\begin{tabular}{l l l l l l l l} \hline \hline
annotations correspond to the intensity scores (ranging from 0 to 1) of several emotions. Thus, the problem can be formulated as a multi-regression problem, where the goal is to predict the intensity scores of each involved emotion. These scores are self-annotated by video subjects, and they relate to seven different emotions: _Adoration_, _Amusement_, _Anxiety_, _Disgust_, _Empathic Pain_, _Fear_, and _Surprise_. This set of emotions may differ from previously predefined sets of basic emotions, e.g., the [PERSON] categorization [52], because they were specifically designed by the dataset authors to better represent the reactions elicited by the video subjects in the dataset. However, the approaches presented in this work can be straightforwardly extended to other emotion types. The Hume-Reaction dataset is notable for its extensive collection of naturalistic emotional reactions. It comprises recordings from 2222 subjects, amounting to over 70 h of data. It also includes audio and video recordings, capturing the subjects' vocal and facial reactions while reacting to an unknown short video clip. All data samples were gathered in an uncontrolled environment, with subjects recording their responses in diverse at-home settings. These settings introduce a variety of noise conditions, making the dataset robust for real-world applications. After viewing each trigger video clip, each recorded subject reported the emotions they experienced and rated the intensity of each emotion. These self-reported data serve as the ground truth for training and evaluating emotion recognition models. For each selected emotion, subjects rated the intensity on a scale from 0 to 1. The dataset is divided into three different splits:  
* A training set made of 15,806 samples from 1334 different human subjects, for a total of 51 h of video recordings. This split was employed in our experimentation to train our models. * A development set made of 4657 samples from 444 different human subjects, for a total of almost 15 h of video recordings. This split was employed in our experimentation to evaluate the proposed approaches. * A private test set made of 4604 samples from 444 different human subjects, for a total of almost 15 h of video recordings. Labels of this split are not publicly available. Thus, this split was not used in this research. A detailed analysis of the dataset's actual scores reveals that the dataset covers the entire spectrum of intensity for each emotion, from 0 to 1. However, there are substantial differences in the average value of the scores for each emotion. Table 2 reports each emotion's average ground truth score. This variability in average scores reflects the diverse emotional expressions and intensities captured in the dataset, adding an additional layer of complexity to the prediction task. Additionally, Figure 1 shows the probability density functions of all emotion scores annotated in the dataset generated by a Gaussian Kernel Density Estimation. They all exhibit a bimodal distribution, featuring significant peaks around zero and one, indicating distinct clusters of low and high intensities within the dataset. In detail, all emotions show a higher peak of around 0, except for amusement and surprises, which have a higher density of around 1.