# HyFusER: Hybrid Multimodal Transformer for Emotion Recognition Using Dual Cross Modal Attention
## 5 Experimental Evaluation
### Experimental Results
This overlap demonstrates that the model struggles to distinguish paralinguistic features such as tone modulation or energy levels, which often blur the line between these emotions. Additionally, 'fear' appears to be underrepresented compared to 'angry' in the dataset, leading to biased feature learning. As a result, the model may prioritize the dominant 'angry' emotion features, inadvertently reducing its ability to recognize 'fear' accurately. Strengthening spectral analysis can help the model better capture subtle differences in vocal patterns. For example, trembling vocal cords associated with 'fear' might display distinct frequency dynamics compared to the sharp pitch shifts of 'angry', allowing the model to separate overlapping paralinguistic cues more effectively. Furthermore, utilizing data augmentation methods to generate synthetic data that mimics the vocal and textual characteristics of 'fear' can mitigate dataset imbalance and provide a more comprehensive representation of underrepresented emotions. By implementing these strategies, the model's ability to differentiate between similar emotions like 'fear' and 'angry' can be significantly improved, reducing misclassifications and enhancing the overall robustness of the emotion recognition system. The Confusion Matrix reveals lower classification accuracy for emotions like 'disgust' and 'fear', with frequent misclassifications such as 'fear' being mistaken for 'angry'. These errors highlight challenges in distinguishing subtle emotional expressions and the limitations of imbalanced datasets. While emotions like 'happy' and 'neutral' benefit from sufficient data representation, underrepresented emotions like 'fear' and 'disgust' suffer from biased learning. Addressing these issues requires enhancing spectral analysis to better capture subtle vocal differences and applying data augmentation methods to generate synthetic examples of underrepresented emotions. These improvements can reduce misclassifications, balance learning across emotions, and enhance the overall robustness and accuracy of the emotion recognition system. Figure 8 provides a comprehensive bar chart visualizing the performance of each multimodal emotion recognition model across four key metrics: Accuracy, Recall, Precision, and F1-Score. The performance values for these four key metrics were derived from Table 5. The models selected for comparison include the average ensemble of single models, the weighted average ensemble of single models, the Bidirectional Cross Modal Attention model, the KoHMT model, and the proposed HyFusER model. Each model is evaluated based on these performance metrics, enabling a direct comparison of their effectiveness. Notably, the HyFusER model achieved the highest scores across all key performance metrics, clearly outperforming existing multimodal models by a significant margin. The success of the HyFusER model is attributed to the effective fusion of text and speech data, advanced feature extraction techniques, and the application of efficient learning mechanisms. These results demonstrate that the HyFusER model represents a substantial advancement in the field of emotion recognition, suggesting its potential applicability in diverse domains such as emotion-based conversational AI, healthcare, and remote education. In conclusion, the performance comparison of multimodal emotion recognition models reaffirms the effectiveness and superiority of multimodal approaches in recognizing and interpreting complex emotional states. This analysis provides essential foundational data for the future development and improvement of emotion recognition technologies.