# Multi-Label Emotion Recognition of Korean Speech Data Using Deep Fusion Models
## 4 Feature Extraction and Model Construction
### Constructing Models for Multi-Label SER
In addition, ViT is especially effective for processing Log-mel spectrograms. Unlike CNNs, which use fixed-size filters, ViT employs a self-attention mechanism that captures both global and local dependencies across the entire image [40]. By dividing the spectrogram into patches and applying self-attention to these patches, ViT can efficiently learn complex patterns that are important information for emotion recognition. Therefore, this study utilizes ViT instead of CNN. Some studies also applied the Mel spectrogram in ViT and demonstrated high performance in SER research [41, 42]. Although ViT models pretrained on ImageNet can be utilized, the goal of this study is to develop a model appropriate for Korean SER. Therefore, instead of using pretrained ViT models, we built the model from scratch. Second, MFCCs is a 1D spectrum feature widely used for vectorizing speech data, representing the energy in frequency bands as coefficients. Therefore, 1D MFCCs can be utilized in the 1D CNN-LSTM architecture for emotion recognition. Figure 6b shows  
Figure 6: (**a**) Vision-Transformer encoder; (**b**) 1D CNN-LSTM structure. the structure of the 1D CNN-LSTM. Specifically, the MFCCs 1D vector is input into the 1D CNN layer, where convolutional filters are applied to detect patterns in the temporal sequence of these MFCCs. The CNN layer operates by sliding these filters across the input vector, capturing localized temporal features that represent variations in speech that correspond to different emotional states. The output from the CNN layer consists of feature maps, which are then downsampled using max pooling to reduce the dimensionality while retaining essential features. These downsampled feature maps are subsequently passed into the LSTM layer, which is designed to capture long-term dependencies and sequential information in the data. Thereby, the LSTM, acting as a decoder in the 1D CNN-LSTM structure, has the advantage of effectively learning in sequential data. As such, utilizing MFCCs with LSTM can facilitate the learning of multiple speech signals, thus enabling the detection of various emotions. By integrating the CNNs in feature extraction and LSTMs in sequence modeling, this architecture enables the effective recognition of emotions from speech. Some studies have utilized MFCCs to the 1D CNN-LSTM and demonstrated high performance in SER research [30, 43]. While this study utilized only LSTM layers for SER due to their effectiveness in capturing long-term dependencies in sequential data, exploring other RNN variants, such as the Content-Adaptive Recurrent Unit (CARU) and Gated Recurrent Unit (GRU), could offer additional advantages in handling complex temporal patterns [44, 45]. Third, the VQFs comprising the 22 acoustic parameters are dimensionless structured data. Therefore, the VQFs can be used in the DNN to classify emotions. The DNN is a neural network comprising multiple layers of perceptron and can learn complex patterns from non-linear data. Hence, the network is trained by inputting 22 acoustic parameter values into the DNN and passing them through several hidden layers to the output layer. In this process, the weights of each layer are adjusted to determine the relationship between the input and output. Finally, after constructing the deep single models, we proceed to construct four deep fusion models using feature fusion: three models by combining features from two single models and one model by combining features from three single models. Specifically, the first fusion model combines ViT and 1D CNN-LSTM, integrating the ability of ViT to capture global patterns in 2D Log-mel spectrograms with the ability of 1D CNN-LSTM to process temporal sequences in MFCCs. The second fusion model integrates ViT with DNN, where ViT focuses on frequency and amplitude variations, and DNN utilizes voice quality characteristics like pitch and intensity. The third fusion model combines 1D CNN-LSTM with DNN, using 1D CNN-LSTM to capture changes over time in MFCCs and DNN to analyze structured voice quality data.