# Multi-Label Emotion Recognition of Korean Speech Data Using Deep Fusion Models
## 6 Concluding Remarks  
Studies on SER using deep learning-based approaches are actively ongoing. However, almost none have specifically used Korean speech data for multi-label classification. Therefore, this study performed multi-label emotion classification using the Korean speech database. To validate this approach, the Log-mel spectrogram, MFCCs, and VQFs were extracted from the Korean speech database provided by the NIA's AI-Hub. Consequently, three deep single and four fusion models were constructed for the SER. In conclusion, the ViT and 1D CNN-LSTM fusion model demonstrated the highest performance in multi-label emotion classification, and achieved an average binary accuracy of 71.2% and an average f1-score of 52.2%. The next highest performing model was the ViT and 1D CNN-LSTM and DNN fusion model, which achieved an average binary accuracy of 70.6% and an average f1-score of 50.9%. This contributes to increasing the efficiency and accuracy of Korean speech multi-label emotion classification. Additionally, this highlights that extracting and utilizing multiple features from speech data is effective for emotion recognition tasks. This study makes the following contributions. First, it contributes to the literature by performing multi-label emotion classification using a Korean speech database. We expect to establish a foundational framework for future studies on Korean speech processing and emotion analysis. Second, these fusion models effectively combine speech features, such as Log-mel spectrogram, MFCCs, and VQFs, to enhance accuracy, rather than relying solely on one speech feature. Thus, we demonstrated that utilizing multiple speech features in combination improves SER performance. Third, we used the AI-Hub speech emotional database, which consists of recordings from a wide range of ages from the 20s to the 50s, without limitations on specific age groups or genders. Therefore, the models constructed using the AI-Hub database can be applied by a diverse range of users in real-world settings where the Korean language is used. Additionally, these models can contribute to a better understanding and management of emotional states by monitoring diverse users. Despite these contributions, further studies are required. First, this study utilized the Log-mel spectrogram, MFCCs, and VQFs to combine various speech features. However, MFCCs are particularly sensitive to background noise, which may affect performance in noisy environments. To address this limitation, future studies should incorporate noise reduction techniques during the preprocessing stage to improve the overall model performance. Next, since ViT has a low inductive bias, its performance can degrade when trained on small-scale datasets. To address this, this study aimed to enhance the model's understanding of local structures and patterns by using rectangle patches instead of squares, thereby improving emotion classification performance. Therefore, with the acquisition of additional data, studies can be carried out to enhance the performance of the ViT model, utilizing both the rectangle patches strategy and the large-scale database. Lastly, while this study did not employ interpretability techniques to analyze the decision-making process, future research will focus on integrating methods such as Grad-CAM or SHAP to improve interpretability. Specifically, by utilizing these advanced techniques, future studies aim to provide deeper insights into the inner workings of the model, offering a more comprehensive understanding of how specific features influence the model's predictions. Conceptualization, S.P., B.J., S.L. and J.Y. ; Methodology, S.P., B.J., S.L. and J.Y. ; Software, S.P. and B.J. ; Validation, J.Y. ; Formal analysis, S.L. ; Investigation, S.P., B.J. and J.Y. ; Resources, J.Y. ; Data curation, S.P. and B.J. ; Writing--original draft, S.P., B.J., S.L. and J.Y. ; Writing--review and editing, J.Y. ; Visualization, S.P. and B.J. ; Supervision, J.Y. ; Project administration, S.P. and J.Y. ; Funding acquisition, J.Y.