# HyFusER: Hybrid Multimodal Transformer for Emotion Recognition Using Dual Cross Modal Attention
## 5 Experimental Evaluation
### Experimental Results
These results highlight its effectiveness in emotion recognition across datasets with different characteristics. The Confusion Matrix analysis for evaluating the emotion recognition performance of the proposed model clearly demonstrates how the model classifies each emotional state. Figure 7 visually presents the accurate classifications and errors made by the model in recognizing seven primary emotional states: 'angry', 'disgust', 'fear', 'happy', 'neutral', \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Model** & **Accuracy (\%)** & \begin{tabular}{c} **Recall** \\ **(Weighted)** \\ \end{tabular} & \
\begin{tabular}{c} **Precision** \\ **(Weighted)** \\ \end{tabular} &
\begin{tabular}{c} **F1-Score** \\ **(Weighted)** \\ \end{tabular} \\ \hline LSTM (Text) & 59.22 & 0.5922 & 0.6278 & 0.6046 \\ CNN (Speech) & 51.85 & 0.5185 & 0.6243 & 0.5584 \\ Single modality & 60.85 & 0.6085 & 0.6660 & 0.6292 \\ average ensemble & & & & \\ Single modality & & & & \\ weighted average & 61.00 & 0.6100 & 0.6638 & 0.6292 \\ ensemble & & & & \\ Bidirectional Cross & 64.08 & 0.6408 & 0.6604 & 0.6419 \\ Modal Attention & 65.62 & 0.6562 & 0.6739 & 0.6628 \\ KoHMT & **67.83** & **0.6783** & **0.6890** & **0.6823** \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Performance evaluation for each model. 'sad', and'surprise'. The diagonal elements of the Confusion Matrix represent the number of correctly classified instances for each emotion. For example, the model correctly classified the 'angry' emotion 154 times and the 'happy' emotion 410 times. These numbers indicate that the model shows relatively high accuracy in recognizing these emotions. In particular, the 'happy' and 'neutral' emotions were the most accurately recognized, possibly due to a sufficient amount of data for these emotions compared to others or because the model effectively learned the features of these two emotions. On the other hand, the off-diagonal elements represent the number of misclassified instances for each emotion. For example, 'angry' was misclassified as 'disgust' 18 times, and 'happy' was misclassified as'surprise' 24 times. These misclassifications suggest that certain emotions have overlapping characteristics or that the model has not sufficiently learned the distinguishing features between some emotions. Additionally, emotions such as 'disgust' and 'fear' showed relatively lower numbers of correct classifications, with only 24 and 78 correctly classified instances, respectively. Furthermore, 'disgust' was misclassified as 'angry' 16 times, and 'fear' was misclassified as 'angry' 54 times. This \begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Model** & **Accuracy (\%)** & **Recall** & **Precision** & **F1-Score** \\  & **(Weighted)** & **(Weighted)** & **(Weighted)** \\ \hline LSTM (Text) & 70.38 & 0.7038 & 0.7086 & 0.7039 \\ CNN (Speech) & 56.38 & 0.5638 & 0.5958 & 0.5703 \\ Single modality & 70.52 & 0.7052 & 0.7232 & 0.7075 \\ average ensemble & & & & \\ Single modality & 71.52 & 0.7152 & 0.7247 & 0.7164 \\ weighted average & & & & \\ ensemble & 77.63 & 0.7763 & 0.7827 & 0.7770 \\ Bidirectional Cross & & & & \\ Modal Attention & 77.45 & 0.7745 & 0.7780 & 0.7744 \\
**HyFusER** & **79.77** & **0.7977** & **0.7975** & **0.7975** \\ \hline \hline \end{tabular}
\end{table} 
Table 6: Comparative experimental performance evaluation. Figure 7: Proposed model Confusion Matrix. indicates that 'disgust' and 'fear' are either less distinct in their expression or that the model has not effectively learned the features necessary to differentiate these emotions. The high confusion between 'fear' and 'angry' emotions, with 54 instances of 'fear' being misclassified as 'angry', highlights the overlapping characteristics of these emotions in vocal and textual features. For instance, trembling or heightened voices associated with 'fear' may be misinterpreted as the aggressive tone characteristic of 'angry'.