# M2 ER: Multimodal Emotion Recognition Based on Multi-Party Dialogue Scenarios
## 3 Evaluation
### Ablation Experiments  
We conducted ablation experiments to validate the effectiveness of different components designed in our multimodal feature fusion model by removing specific modules in the multimodal fusion part.  
Our fusion model mainly consists of two parts: _Part 1_ utilizes cross-modal attention to capture the interaction between modalities; _Part 2_ utilizes the attention mechanism to determine the importance of each modality for the final classification and fuses the multimodal information. We conducted comparative experiments between direct concatenation and the fusion mechanism we adopted, as shown in Table 7.  
Comparing the experimental results in Figure 13, _Fusion 2_ improved the performance by 1.09% compared to _Fusion 1_. Compared with _Fusion 1_ and _Fusion 2_, _Fusion 3_ improved the performance by 3.4% and 2.3%, respectively, as _Fusion 2_ and _Fusion 3_ utilize the correlated information between modalities and effectively allocate importance weights to each modality. The ablation experiments confirmed the effectiveness of each module in the fusion model in our work--_Part 1_ and _Part 2_.  
\begin{table}
\begin{tabular}{c c c c c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**Auger**} & \multicolumn{5}{c}{**Emotion**} \\ \cline{3-8}  & & **Anger** & **Joy** & **Neutral** & **Sadness** & **Surprise** & **w-avg F1** \\ \hline bclSTM & text + audio & 45.9 & 52.2 & 77.9 & 11.2 & 49.9 & 60.6 \\ \hline \multirow{3}{*}{DialogueRNN} & text & 41.7 & 53.7 & 77.8 & 21.2 & 47.7 & 60.8 \\  & audio & 34.1 & 18.8 & 66.2 & 16 & 16.6 & 44.3 \\  & text + audio & 48.2 & 53.2 & 77.7 & 20.3 & 48.5 & 61.6 \\ \hline \multirow{3}{*}{M2 ER} & text & 42.1 & 53.2 & 78.6 & 35.9 & 52.3 & 62.9 \\  & audio & 32.5 & 31.0 & 66.1 & 13.6 & 23.2 & 46.7 \\ \cline{1-1}  & video & 29.2 & 26.0 & 65.7 & 19.5 & 29.6 & 46.3 \\ \cline{1-1}  & text + audio + video & 45.6 & 55.1 & 79.4 & 36.3 & 53.9 & 64.3 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Scores for the five emotion classifications on the test set.  
Figure 12: Results of unimodal and multimodal emotion classification of our model on the test set. They present a comparison of the w-avg F1 values for both unimodal and multimodal emotion recognition.  
Since the baseline models on MELD did not utilize the video modality, and it was found that most dialogue emotion recognition studies based on MELD also did not utilize the video modality through research, A comparative analysis was performed between the video modality emotion recognition models without speaker recognition and the models utilizing it in order to validate the effectiveness of our proposed speaker recognition method. The results are presented in Table 8.  
By comparing the data in Table 8, it can be observed that our model achieved a w-avg F1 score of 42.73% for emotion classification in the video modality. Furthermore, by comparing the data of _video_' and _video_, it is evident that our proposed speaker recognition method improved the emotion recognition performance in the video modality by 6.58%. The comparison in Table 8 indicates that the method effectively enhances the efficiency of extracting emotional features in multi-party dialogue scenarios, highlighting the role of the video modality in emotion recognition during multi-party dialogues.