# Multi-Label Emotion Recognition of Korean Speech Data Using Deep Fusion Models
## 2 Related Works  
In recent years, studies on SER have emerged as a crucial area in HCI and speech processing [23]. Consequently, studies on SER using large-scale data are being conducted, aiming to apply deep learning-based methods to recognize various emotions [24]. Deep learning-based SER can automatically recognize human emotional states through HCI technology, enabling the provision of customized services that reflect individual moods. In addition, speech-based mental health monitoring can also contribute to the detection of stress or depression in users. To this end, SER studies have mainly focused on multi-class and multi-label classification. First, most previous studies on SER were performed for multi-class classification, aimed at recognizing one emotion among many, using a single speech feature. For example, one study utilized segmented speech utterance data in a Deep Neural Network (DNN) to classify emotions such as Anger, Neutral, and Sadness [25]. Additionally, spectrograms extracted from the EmoDB dataset and applied them to a deep Convolutional Neural Network (CNN) for classify emotion recognition such as anger, boredom, disgust, fear, happiness, neutral, and sadness [26]. Furthermore, MFCCs extracted from the RAVDESS dataset were used in a Long Short-Term Memory (LSTM) to identify emotions such as calm, happiness, sadness, anger, fearful, surprise, and disgust [16]. Moreover, raw audio clips and Log-mel spectrogram extracted from the EmoDB and IEMOCAP datasets were utilized in 1D and 2D CNN-LSTM, respectively, to recognize emotions [14]. Additionally, a study was conducted on emotion recognition using Bi-LSTM with the Log-mel spectrogram extracted from the EmoDB and IEMOCAP datasets [13]. These studies have made significant contributions to deep learning-based SER by focusing on the use of a single speech feature, such as Mel spectrograms or MFCCs. While this approach effectively identifies distinct emotions, relying on only one speech feature may limit the ability to fully capture the complexity of emotional expressions. To improve emotion recognition performance, some previous studies on SER aimed to enhance the accuracy of recognizing by using multiple speech features. For example, one study proposed the application of a Dual-Sequence LSTM, an extension of LSTM, utilizing MFCCs and Mel spectrograms for recognizing emotions such as Happiness, Neutral, Anger, and Sadness, extracted from the IEMOCAP dataset [27]. Additionally, a study was conducted to improve the performance in emotion recognition by constructing a fusion model that combines Bi-LSTM and CNN, utilizing MFCCs and ERB spectrograms extracted from a Korean speech database [4]. These studies have contributed to enhancing the performance of SER by incorporating multiple speech features, such as MFCCs and Mel spectrograms. However, despite the improved performance, these studies primarily focused on multi-class classification and did not address the challenge of multi-label classification, which is essential for recognizing mixed emotions that often occur in the real-world scenarios. Recognizing the importance of addressing mixed emotions, some previous studies on SER were performed for multi-label classification. For example, to enhance the performance of emotion recognition, speech utterances were extracted from the IEMOCAP dataset and applied to a Response Residual Network to recognize mixed emotions such as a combination of happiness, neutral, sadness, and anger [17]. Additionally, a study was conducted to improve performance in recognizing emotions such as a combination of disgust, happiness, fear, anger, surprise, and sadness by constructing a fusion model of CNN and LSTM, utilizing Log-mel spectrograms and MFCCs extracted from the RML dataset [18]. These studies have contributed to the literature as the first to conduct multi-label classification using English or German speech data. However, an important gap in the research is the lack of focus on multi-label classification using languages other than English or German, such as Korean.