# M2 ER: Multimodal Emotion Recognition Based on Multi-Party Dialogue Scenarios
## 3 Evaluation
### Model Scalability Verification  
To validate the scalability of our multimodal emotion recognition model, we conducted emotion recognition experiments on the MEISD dataset, which is also a multi-party dialogue dataset. Additionally, we performed emotion recognition tests on some real-world \begin{table}
\begin{tabular}{c c c c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{8}{c}{**Emotion**} \\ \cline{3-10}  & **Anger** & **Disgust** & **Fear** & **Joy** & **Neutral** & **Sadness** & **Surprise** & **w-avg F1** \\ \hline \multirow{3}{*}{M2 ER} & Fusion 1 & text + audio + video & 41.81 & 11.63 & 7.10 & 54.37 & 77.84 & 40.09 & 53.41 & 60.74 \\  & Fusion 2 & text + audio + video & 41.99 & 15.83 & 7.90 & 55.07 & 78.52 & 40.39 & 53.86 & 61.40 \\  & Fusion 3 & text + audio + video & 43.75 & 16.93 & 9.62 & 56.63 & 80.11 & 41.67 & 54.14 & 62.83 \\ \hline \hline \end{tabular}
\end{table} 
Table 7: Results of ablation experiments for multimodal fusion emotion recognition. _Fusion 1_ represents direct concatenation; _Fusion 2_ represents the first part of the fusion model, _Part 1_, which considers only the interaction between modalities and within each modality; _Fusion 3_ represents the complete fusion model, _Part 1 + Part 2_. This table has been transformed into a bar chart, as shown in Figure 13.  
Figure 13: Experimental results of multimodal emotion recognition ablation. They present the comparison of the w-avg F1 scores for different variants of the multimodal fusion model on MELD. \begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{8}{c}{**Emotion**} \\ \cline{2-9}  & **Anger** & **Disgust** & **Fear** & **Joy** & **Neutral** & **Sadness** & **Surprise** & **w-avg F1** \\ \hline video\({}^{\prime}\) & 22.64 & 5.41 & 3.32 & 23.07 & 60.34 & 14.15 & 24.16 & 40.05 \\ video & 24.43 & 6.62 & 4.54 & 22.89 & 63.68 & 20.07 & 29.44 & 42.73 \\ \hline \hline \end{tabular}
\end{table} 
Table 8: Results of the ablation experiments in the video modality. _video_’ represents the method where our method was not used. _video_ represents our proposed method of using speaker recognition. data using the fusion model and presented the test results to visualize the generalization performance of the model. We further conducted multi-class emotion recognition experiments on the MEISD dataset, using the micro-F1 score as the evaluation metric. Since the distribution of each emotion label in the training, validation, and test sets of the MEISD dataset is relatively balanced, the prediction performance for each emotion label is almost the same. Therefore, we collected the overall w-avg F1 score for comparison and presentation in the experiments. The results are shown in Table 9.  
Figure 14 shows that our multimodal emotion recognition approach also performs well on the MEISD dataset. The w-avg F1 score for text modality emotion recognition is 61.10%, for audio modality is 43.49%, and for the video modality is 42.91%. The performance of individual modalities in emotion recognition surpasses that of classical baseline models. The fusion model achieves a w-avg F1 score of 62.97%, outperforming the individual modality recognition results. These experimental results demonstrate the effectiveness and scalability of our multimodal emotion recognition model in multi-party dialogue scenarios. To better illustrate the scalability of the model, we tested it on some actual examples, as shown in Table 10. From the table, it can be observed that in the case of Example A, the facial image of the speaker obtained from the video modality shows an upward curvature of the mouth, indicating a smiling expression. Additionally, the speaker's voice has a high pitch and a cheerful speaking rate. The text also expresses a positive emotion, leading to  
Figure 14: Emotional classification results of the MEISD. They present the comparison of w-avg F1 scores for unimodal and multimodal emotion recognition of various models on the MEISD dataset.