# M2 ER: Multimodal Emotion Recognition Based on Multi-Party Dialogue Scenarios
## 3 Evaluation
### Results of Multimodal Experiments  
Relevant experiments were conducted on the training, validation, and test sets of MELD. The experimental results are shown in Table 5. This model utilizes cross-modal attention interaction to capture correlated information between modalities, obtaining feature representations with interactive effects. By using the attention mechanism, it determines the importance of each modality in the final fusion classification and combines the multimodal information.  
Comparing the results in Figure 12 shows that our three-modality fusion emotion recognition model achieved a w-avg F1 score of 62.83%, outperforming the individual modalities of text, audio, and video in emotion recognition. Compared to unimodal data, multimodal data can capture more diverse emotional features. Multimodal fusion can also compensate for the limitations of individual modalities. Furthermore, our multimodal fusion model shows improved fusion performance compared to several baseline models. It was found that the recognition results for disgust and fear were not satisfactory. To address this, we further conducted a five-class emotion recognition experiment on MELD, excluding the less frequent emotions of fear and disgust. The results of this experiment are shown in Table 6.  
Figure 11: Performance results of different models of text and audio modalities on MELD. The figure shows the comparison of the w-avg F1 values for different models in text and audio modality emotion recognition on MELD.  
\begin{table}
\begin{tabular}{c c c c c c c c c} \hline \hline \multicolumn{2}{c}{**Model**} & \multicolumn{6}{c}{**Emotion**} \\ \cline{3-10}  & & **Anger** & **Disgust** & **Fear** & **Joy** & **Neutral** & **Sadness** & **Surprise** & **w-avg F1** \\ \hline bcLSTM & text + audio & 43.39 & 23.66 & 9.38 & 54.48 & 76.67 & 24.34 & 51.04 & 59.25 \\ \hline DialogueRNN & text + audio & 43.65 & 7.89 & 11.68 & 54.40 & 77.44 & 34.59 & 52.51 & 60.25 \\ \hline \multirow{4}{*}{M2 ER} & text & 40.45 & 15.24 & 5.55 & 53.31 & 77.57 & 37.85 & 52.42 & 60.05 \\  & audio & 31.51 & 9.02 & 5.25 & 29.08 & 64.84 & 13.06 & 20.13 & 43.39 \\ \cline{1-1}  & video & 24.43 & 6.62 & 4.54 & 24.89 & 63.68 & 20.07 & 29.44 & 42.73 \\ \cline{1-1}  & text + audio + video & 43.75 & 16.93 & 9.62 & 56.63 & 80.11 & 41.67 & 54.14 & 62.83 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Scores for the seven emotion classifications on the test set. _“text + audio + video”_ represents our attention-based multimodal fusion model. This table was transformed into a bar chart, as shown in Figure 12.  
By comparing the results in Tables 5 and 6, it can be observed that after excluding two less frequent emotions, the five-class emotion recognition performance significantly improved compared to the seven-class classification because there are very few samples for the emotions of fear and disgust in the training set. Additionally, distinguishing between anger, disgust, and fear is challenging as the differences between these emotions are subtle. This explains why the recognition results for disgust and fear were relatively poor in the seven-class emotion recognition experiment. Furthermore, the performance of the text modality in emotion recognition remained generally superior to that of the audio and video modalities in the five-class emotion recognition experiment, and the multimodal emotion recognition outperformed the single modality.