# Multi-Label Emotion Recognition of Korean Speech Data Using Deep Fusion Models
###### Abstract  
As speech is the most natural way for humans to express emotions, studies on Speech Emotion Recognition (SER) have been conducted in various ways However, there are some areas for improvement in previous SER studies: (1) while some studies have performed multi-label classification, almost none have specifically utilized Korean speech data; (2) most studies have not utilized multiple features in combination for emotion recognition. Therefore, this study proposes deep fusion models for multi-label emotion classification using Korean speech data and follows four steps: (1) preprocessing speech data labeled with Sadness, Happiness, Neutral, Anger, and Disgust; (2) applying data augmentation to address the data imbalance and extracting speech features, including the Log-mel spectrogram, Mel-Frequency Cepstral Coefficients (MFCCs), and Voice Quality Features; (3) constructing models using deep fusion architectures; and (4) validating the performance of the constructed models. The experimental results demonstrated that the proposed model, which utilizes the Log-mel spectrogram and MFCCs with a fusion of Vision-Transformer and 1D Convolutional Neural Network-Long Short-Term Memory, achieved the highest average binary accuracy of 71.2% for multi-label classification, outperforming other baseline models. Consequently, this study anticipates that the proposed model will find application based on Korean speech, specifically mental healthcare and smart service systems. deep fusion model; multi-label classification; speech emotion recognition +
Footnote †: journal: Medical  
focuses on recognizing both single and mixed emotions, considering the possibility of multiple emotions being derived in combination [11]. First, most SER studies have been performed based on multi-class classification which utilizes various speech features. For example, a study was conducted using voice quality parameters, which represent the pitch, loudness, and speech clarity from the EmoDB dataset built in German for SER [12]. Additionally, other studies have been conducted using the Log-mel spectrogram, which represents the frequency, amplitude, and time domain of speech, with data from the EmoDB dataset [13,14]. Another SER study was conducted using Mel-Frequency Cepstral Coefficients (MFCCs), which represent the unique speech information in the frequency domain, from the RAVDESS dataset built in English for SER [15,16]. Second, some SER studies have been performed based on multi-label classification. For example, a study was conducted using speech utterances from the IEMOCAP dataset built in English for SER [17]. Additionally, a study was conducted using the Log-mel spectrogram and MFCCs in combination to perform multi-label emotion classification on RML datasets, including various languages such as English, Italian, Persian, and Chinese [18]. Despite the significant contributions of previous studies on SER, some areas still need improvement. First, while most studies have focused on multi-label emotion classification with English and German speech data, relatively few have been on Korean speech data. Since there are differences in speech signals such as speech rate, intonation, and morphemes in the expression of emotions depending on the culture and language of each country [19], constructing a model using speech databases of each country's language is imperative. Therefore, further studies are required to propose a multi-label classification model that accurately reflects the diverse characteristics of Korean speech. Second, most previous studies have utilized a single speech feature, such as the Log-mel spectrogram or MFCCs to recognize emotions, rather than combinations of different speech features. Although a single feature can be used to recognize emotions, combining and using multiple features extracted from speech signals further improves the accuracy and objectivity of emotion recognition [4,20].