# Emotion Recognition from Videos Using Multimodal Large Language Models
## 5 Experimental Results
### Qualitative Analysis  
To better understand the performance differences between the ViPER [9] models based on CLIP and those based on Video-LLaVA [12], we conducted a qualitative analysis by discretizing the predictions and ground truth (GT) into bins of 0.1 range. This was done separately for each emotion. The confusion matrices reveal how often predictions fell within certain ranges of the true emotion scores. Examining these matrices allows us to observe prediction patterns and identify areas where each approach excelled or fell short.  
A key observation from the confusion matrices is the range of prediction values:  
* ViPER-VATF\({}_{CLIP}\): The predictions are more concentrated near the GT average value. This indicates that the CLIP-based solution is more focused on predicting scores that are close to the average GT value. It suggests a tendency to overfit on the average value, making it more accurate for samples whose scores are near the mean.
* ViPER-VATF\({}_{Video-LLaVA}\): The predictions cover a wider range of values. This means that the LLM-based solution is better at predicting scores that can be considered outliers with respect to the average value. These outliers include cases where an emotion is particularly evident or notably missing.  
Figures 5 and 6 show the confusion matrices obtained using the original ViPER-VATF [9] architecture for the Adoration and Empathic Pain emotions, respectively. Notably, the predictions focus in the range \([0.2,0.5]\) for Adoration and \([0.1,0.4]\) for Empathic Pain, with just a few predictions in the adjacent bins. Figures 7 and 8 show the confusion matrices obtained by integrating the textual features from Video-LLaVA [12] into the ViPER-VATF [9] architecture for the same emotions. It can be observed that the majority of predictions fall in a range of values wider with respect to the previous case, i.e., \([0.1,0.6]\) for Adoration and \([0.0,0.5]\) for Empathic Pain.  
Figure 5: ViPER-VATF [9] based on CLIP [26] textual features prediction for the Adoration emotion. The average prediction is \(0.3737\pm 0.0867\), with a minimum prediction score of 0 and a maximum of 0.5500.  
Figure 6: ViPER-VATF [9] based on CLIP [26] textual features prediction for the Empathic Paining emotion. The average prediction is \(0.2088\pm 0.0669\), with a minimum prediction score of \(0.0812\) and a maximum of \(0.5513\).  
Figure 7: ViPER-VATF [9] based on Video-LLaVA [12] textual features prediction for the Adoration emotion. The average prediction is \(0.3465\pm 0.1387\), with a minimum score of \(0\) and maximum score of \(0.6289\).  
The wider range of predictions in the Video-LLaVA-based approach suggests its superior ability to detect and accurately score emotions that deviate significantly from the mean. This capability is particularly valuable in scenarios where certain emotions are either strongly expressed or barely present, which is often critical for nuanced emotional understanding.  
In summary, while the CLIP-based solution shows robustness in predicting common emotional expressions, the LLM-based solution offers a more comprehensive approach capable of recognizing and scoring a wider variety of emotional intensities, including extreme cases. This qualitative analysis underscores the potential for combining both approaches to achieve a more balanced and accurate emotion recognition system.