# Multi-Label Emotion Recognition of Korean Speech Data Using Deep Fusion Models
## 5 Results and Performance Analysis
Finally, the emotions were classified using MLP, with dropout regularization applied to the fully connected layer to avoid overfitting. The architectures of the 1D CNN-LSTM and DNN are shown in Figures 9 and 10. In Figure 9, the 1D CNN acts as an encoder in the 1D CNN-LSTM architecture, comprising three layers. During the convolution operation, the number of filters and filter lengths were set to (32, 3), (64, 3), and (128, 3), respectively. Consequently, the input MFCCs vector sized (300, 40) underwent a convolution operation and max pooling to transform it into a feature vector sized (35, 128). The LSTM layer used this feature vector from the 1D CNN as input and converted it into a vector sized (1, 15). Subsequently, a fully connected layer with a dropout was employed. Finally, in Figure 10, the first hidden layer of the DNN model has 32 neurons, the second hidden layer has 16 neurons, and the output layer has 5 neurons. Between these layers, the GELU activation function, batch normalization, and dropout were applied to avoid overfitting. Furthermore, for model training, the Adam optimizer was employed, selected for its computational efficiency and low memory requirements. The Adam optimizer's learning rate was set within the range of [0.0005, 0.001]. Additionally, the hyperparameters were defined with epochs in the range [50, 150], the batch sizes ranged [32, 256], the early stopping patience ranged [5, 8], and gamma values within the range [0.5, 2.5]. We identified and utilized the hyperparameters that exhibited optimal performance throughout the model training and performance evaluation processes. Finally, the models were trained and evaluated with the sigmoid function threshold set to 0.45, which outputs the probability values for the five classes at the final output layer. This threshold was chosen instead of the standard of 0.5 because it led to higher performance by enabling a more complex combination of various emotions to be derived. The overall performances of the single deep models constructed in this study are listed in Table 8. The performances of the fusion models are listed in Table 9.  
Figure 8: Vision-Transformer model architecture with rectangle patches. As Table 8 shows, both the ViT and 1D CNN-LSTM models exhibited notably higher performance than the DNN model. Specifically, the average binary accuracy was the highest for the 1D CNN-LSTM model at 70.7%, whereas the average f1-score was the highest for the ViT model at 51.2%. However, the DNN model utilizing VQFs did not exhibit high performance in the multi-label emotion classification, particularly showing relatively low recall and f1-score for the 'Anger' emotion compared to other emotions. This indicates that VQFs do not significantly contribute to identifying the 'Anger' emotion. However, the ViT model utilizing the Log-mel spectrogram and the 1D CNN-LSTM model utilizing MFCCs demonstrated relatively higher performance across all emotions, suggesting that these two speech features are effective for SER. Figure 10: DNN model architecture. Figure 9: The 1D CNN-LSTM model architecture.