# Multi-Label Emotion Recognition of Korean Speech Data Using Deep Fusion Models
## 5 Results and Performance Analysis
Indeed, while the other three fusion models exhibited enhancements in performance compared to the DNN model, they did not surpass those of the ViT and 1D CNN-LSTM fusion models. This indicates that the lower performance of the DNN model compared to the other single models contributed to a decrease in the overall performance when integrated into the fusion models. Next, having the highest average binary accuracy and average f1-score indicates that the ViT and 1D CNN-LSTM fusion model more accurately identifies the multi-label classifications of mixed emotions compared to other models. The architecture of the ViT and 1D CNN-LSTM fusion model is shown in Figure 11. In Figure 11, the feature fusion approach was applied, where the fully connected layers derived from ViT and 1D CNN-LSTM are concatenated and processed through an MLP to perform multi-label classification. The other fusion models have similar architectures, as shown in Figure 11. The other fusion models involve extracting features from single models, concatenating these features, and subsequently using an MLP for the final classification task. This approach allows for the integration of diverse feature representations, enabling the model to better capture the mixed emotions in speech data. To further explore the model's performance, this study conducts an in-depth analysis of the ViT and 1D CNN-LSTM fusion model, which has shown the best performance among the constructed models. When examining the model's performance in more detail, the  
Figure 11: Vision-Transformer and 1D CNN-LSTM fusion model architecture. performance metrics reveal a higher binary accuracy, suggesting that the model is generally effective at distinguishing between the presence and absence of emotions. However, this model shows a lower f1-score and recall compared to the binary accuracy. Specifically, the model's ability to achieve a higher True Negative Rate (TNR) across all emotions highlights its effectiveness in minimizing false positives. The lower recall and f1-score imply that, while the model performs well in minimizing false positives, challenges arise in correctly identifying all classes of certain emotions, particularly when those emotions are less distinctly expressed or when multiple emotions are mixed in a speech recording. For instance, 'Anger' and 'Disgust' exhibit relatively lower recall values compared to other emotions. This issue is a common challenge in multi-label emotion recognition, where the complexity of the task often leads to imbalances in precision and recall. Despite these challenges, the ViT and 1D CNN-LSTM fusion model demonstrates reliable overall performance. With further improvements, the model has the potential to achieve even better results, enhancing its ability to accurately recognize a wide range of emotions. Conclusively, by utilizing the Korean emotional speech database provided by the AI-Hub for constructing multi-label emotion classification, this study demonstrated that the ViT and 1D CNN-LSTM fusion model is the most appropriate choice. While deep single models constructed using either Log-mel spectrogram or MFCCs also demonstrated commendable performance, the fusion models that utilized both speech features generally evidenced better performance. This implies that integrating multiple speech features into SER is more effective than using a single feature. However, studies on multi-label emotion recognition are limited, making direct performance comparisons with prior studies challenging. Specifically, recent studies exploring multi-label emotion classification often employ multimodal approaches, where different modalities such as facial expressions or text are combined with speech to recognize emotions [51; 52]. This multimodal approach in recent studies complicates direct comparisons with our method, which relies solely on the speech modality.