# Emotion Recognition from Videos Using Multimodal Large Language Models
## 5 Experimental Results
### Quantitative Results  
Table 3 presents the performance of the proposed approaches for video emotion recognition, evaluated using the mean Pearson correlation [53] among all involved emotions. The first half of the table reports the dataset author's baselines and original ViPER [9] results. The second half shows our querying, probing, and integration results exploiting video and visual LLMs. The proposed methods involving Video-LLaVA [12] and LLaVA [11] show varying levels of effectiveness:  
* **Querying Video-LLaVA**: Directly querying Video-LLaVA [12] for emotion scores resulted in a mean Pearson correlation of 0.0937, which is lower than all baselines, indicating limited effectiveness for this approach. Additionally, it was observed that the generated text often used the same exact score value or a limited range of values for some emotions, e.g., the score 0.4 appears 2444 times out of 4657 in the Anxiety predictions. This suggests that text generation in a zero-shot fashion may be unsuitable for regression tasks, as it lacks the precision required for accurate scoring across a continuous range. * **Probing Video-LLaVA**: Fine-tuning with probing strategies showed improvements, with mean Pearson correlations of 0.2333 for Prompt 1 and 0.2351 for Prompt 2. Although these scores did not surpass the baselines, they highlighted the potential of probing strategies. Additionally, this result indicates that the prompts used, whether general or specific for emotion recognition, do not greatly impact the performance. We also studied the impact of the employed activation function within the probing network. Table 4 reports the results obtained using seven different activation functions while adopting Prompt 2. Noteworthy is that the variation in performance as the activation function varied was very limited, i.e., from 0.2315 to 0.2353. However, the ReLU-based functions achieved a slightly superior result. * **Integrating Video-LLaVA textual features**: Integrating Video-LLaVA-generated textual features into the ViPER-VATF [9] framework showed competitive performance, with mean Pearson correlations of 0.3004 for the general prompt and 0.3011 for the specific prompt, closely matching the performance of the original ViPER-VATF [9]. If the results are broken down by observing each emotion separately, these approaches surpassed the classical ViPER-VATF [9] for specific emotions. Specifically, using Prompt 1 yielded better performance for Anxiety and Empathic Pain, while Prompt 2 performed better on Adoration, Anxiety, and Surprise. On the other hand, the CLIP-based approach achieved the highest results in Amusement, Disgust, and Fear. Table 5 reports the breakdown results. Furthermore, we compared the impact that textual, acoustic and FAUs features had when combined with visual ones. The results are reported in Table 6. It is important to note that the textual features extracted from Video-LLaVA [12], although we did not have a contribution at the level of the FAU features, alwaysbrought a benefit when injected into the model; this is in contrast to the acoustic ones, which occasionally did not improve or even worsened the performance of the model. * **Integrating LLaVA textual features**: Using LLaVA [11] to describe video frames separately and integrating these frame-specific textual features into the ViPER-VATF [9] framework resulted in a mean Pearson correlation of 0.2895, indicating the viability of this alternative approach. However, integrating Video-LLaVA [12] textual was still better (up to 0.3011 vs. 0.2895). \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Image** & **Audio** & **Text** & **FAU** & **Mean Pearson Correlation** \\ \hline ✓ & - & - & - & 0.2712 \\ ✓ & ✓ & - & - & 0.2748 \\ ✓ & - & ✓ & - & 0.2758 \\ ✓ & - & - & ✓ & 0.2978 \\ ✓ & ✓ & ✓ & - & 0.2758 \\ ✓ & - & ✓ & ✓ & **0.3011** \\ ✓ & ✓ & - & ✓ & 0.2924 \\ ✓ & ✓ & ✓ & ✓ & **0.3011** \\ ✓ & ✓ & - & ✓ & 0.2924 \\ ✓ & ✓ & ✓ & ✓ & **0.3011** \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Ablation study on different modalities.