# Multi-Label Emotion Recognition of Korean Speech Data Using Deep Fusion Models
## 4 Feature Extraction and Model Construction
### Extracting Speech Features  
In speech analysis, the two primary methods utilized consist of converting speech data into time and frequency domains, or raw speech signals. Specifically, the speech features utilized in speech analysis include MFCCs and Mel spectrogram, which exploit the characteristics of the speech signals in the frequency or time domains, and VQFs, which represent the raw signals and quality of the speech itself. First, the Mel spectrogram is a feature extraction technique used in speech analysis to capture changes in frequency and amplitude characteristics over time. Humans perceive low-frequency sounds as more sensitive, and sounds become less sensitive as their frequency increases. Expressed differently, humans do not perceive frequency linearly, but rather through the Mel-scale. This non-linear perception of frequency was modeled using the Mel-scale function, which reflects the criteria for human speech perception in terms of frequency, considering the characteristics of the human cochlea. The Mel-scale function, which converts Hertz to Mel, is represented by Equation (1), where **M** represents Mel, **f** represents Hertz, and the constant **2595** is used to convert the linear frequency to the Mel scale, reflecting the non-linear way humans perceive pitch [33]. \[M=2595\log\left(1+\frac{f}{700}\right) \tag{1}\]  
Therefore, to extract the Log-mel spectrogram, the spectrogram obtained through Short-Time Fourier Transform (STFT) is mapped to the Mel-scale function or a Mel-filterbank to convert it into the Mel spectrogram. Then, the logarithm for each value of the Mel spectrogram is then taken. This process produces the Log-mel spectrogram, which distributes energy across a wider frequency range, enabling more detailed feature extraction. Second, MFCCs are derived by applying a discrete cosine transform to the Mel spectrogram to extract distinctive speech features [34]. The MFCCs represent unique feature vectors extracted from speech, capturing the energy distribution across frequency bands through a set of coefficients. The primary objective of extracting MFCCs is to efficiently capture various characteristics of speech data in vector form. The MFCCs extraction function is represented by Equation (2), where \(\hat{S}_{k}\) denotes the vector values obtained through the Mel-scale process, **k** represents the index of the filterbank, and **n** indicates the number of MFCCs coefficients to be extracted, where n = 1, 2...k [35]. \[C_{n}=\sum_{k=1}^{k}\ \left(log\hat{S}_{k}\right)cos\left[n\left(k-\frac{1}{2} \right)\frac{\pi}{k}\right] \tag{2}\]  
In speech analysis, the Log-mel spectrogram and MFCCs represent heterogeneous data with different speech features. These are vectorized and stored separately to utilize in combination. Figure 5 shows a sample visualization of the Log-mel spectrogram and the MFCCs generated using the librosa library. Third, VQFs are extracted to represent the quality and signals of the speech itself. Since human voices lack regularity, the acoustic parameters that measure irregularity, fluctuations, and noise in the waveform, referred to as VQFs, are often used to enhance the performance of speech recognition systems. Therefore, 22 acoustic parameters related to VQFs are extracted for analysis. Specifically, the acoustic parameters related to fundamental frequency, which signifies the lowest frequency of a periodic waveform (F0 mean, F0 stdev); parameters related to the ratio of harmonics to noise energy (Harmonics-to-Noise Ratio); parameters related to frequency variation or frequency perturbation (Local jitter, Local absolute jitter, Rap jitter, PPQ5 jitter, DDP jitter); and parameters related to amplitude variation (Local shimmer, Local dB shimmer, APQ3 shimmer, APQ5 shimmer, APQ1 shimmer, DDA shimmer) are extracted from the speech periodic wave.