# Multi-Label Emotion Recognition of Korean Speech Data Using Deep Fusion Models
## 4 Feature Extraction and Model Construction
### Constructing Models for Multi-Label SER  
This section describes the construction of three deep single models and four deep fusion models. The Log-mel spectrogram is a two-dimensional (2D) form of data that represents the change in frequency and amplitude over time. Therefore, the 2D Log-mel spectrogram images can be used in the ViT architecture. Specifically, ViT divides input images into patches and performs operations patch-by-patch. The ViT architecture comprises a patch embedding, an encoder, and the Multi-Layer Perceptron (MLP) classifier. Initially, the image is tokenized by dividing it into patches. Then, the linear embedding is applied to each patch to make it a sequence, which is input to the transformer encoder and the encoder generates features by the correlation between each patch through the self-attention mechanism, and finally generated features are classified by the classifier [36]. By dividing the image into multiple patches, local features can be better extracted, thereby improving the efficiency and accuracy of image processing [37]. In this patch division process, the image can be divided into patches of squares, rectangles, or triangles based on the features of the image [37; 38; 39]. Although conducting self-attention in square patches may be optimal, this study divides the Log-mel spectrogram into rectangular patches to analyze the original image without resizing it for compaction or extension. Resizing might be necessary to create square patches, especially when considering rectangular images. Figure 5: Example of audio data converting to the Log-mel spectrogram and MFCCs. \begin{table}
\begin{tabular}{c c} \hline \hline
**Main Features** & **Derivative Features** \\ \hline Fundamental frequency (F0) & F0 mean, F0 stdev \\ \hline Harmonics-to-Noise Ratio (HNR) & HNR \\ \hline Jitter & Local jitter, Local absolute jitter, Rap jitter, \\  & PPQ5 jitter, DDP jitter \\ \hline Shimmer & Local shimmer, Local dB shimmer, APQ3 shimmer, \\  & APQ5 shimmer, APQ11 shimmer, DDA shimmer \\ \hline Intensity & Intensity max, Intensity mean, Intensity min, \\  & Intensity dynamic range, Intonation variation \\ \hline Pitch & Pitch max, Pitch mean, Pitch range \\ \hline \hline \end{tabular}
\end{table} 
Table 5: Description of the 22 acoustic parameters. This decision is based on the understanding that speech is characterized by sequences and the Log-mel spectrogram represents frequency and amplitude at specific time intervals. Therefore, dividing the patches into rectangles along the time axis can capture changes in frequency and amplitude over short periods, thereby enabling the identification of evolving emotions. Once the image patches are extracted, linear embedding is applied to each token and then input to the encoder. Figure 6a shows the ViT encoder structure. The encoder mainly comprises Layer Normalization (LN), Multi-Head Self-Attention (MHSA), MLP, and residual connection, and is calculated using Equations (3) and (4) [36]. \[Z_{l}^{\prime}=MHSA(LN(Z_{l-1}))+Z_{l-1} \tag{3}\]  
\[Z_{l}=MLP\big{(}LN\big{(}Z_{l}^{\prime}\big{)}\big{)}+Z_{l}^{\prime} \tag{4}\]  
In Equation (3), \(Z_{l}^{\prime}\) represents the result of the LN, MHSA, and residual connection applied to the embedding vector \(\mathbf{Z}_{l-1}\), which is used as the input to the encoder. In Equation (4), \(\mathbf{Z}_{l}\) represents the result of LN, MLP, and residual connections performed on the output of Equation (3). The processes described in Equations (3) and (4) are repeated for each layer of the encoder. Through this repetition, the patch embeddings of the input images are transformed into more complex representations of the underlying relationships. By combining and updating the embedding information for each patch and the entire images, the model can learn numerous parameters to classify the images effectively. Therefore, the Log-mel spectrogram can be utilized in ViT to recognize both the global features of the speech and the detailed changes and patterns occurring in specific frequency bands.