# Speech Emotion Recognition Based on Two-Stream Deep Learning Model Using Korean Audio Information
## 3 Results
### Results and Performance Analysis  
This study comparatively analyzed the performance of a two-stream-based SER model using the Bi-LSTM and CNN two-stream-based transfer learning model, as well as various other deep learning models, by extracting multidimensional features of speech data. Table 1 overviews the experimental environment. The first experiment was conducted for emotion recognition based on the LSTM and Bi-LSTM using the spectrum features of the Korean speech data. Three different feature extraction methods were used for the LSTM and Bi-LSTM models to compare the emotion recognition performance. Table 2 lists the learning parameters used for the models. Figure 7: Visualization of speech data for eight emotions. Table 3 compares the performances of the LSTM and Bi-LSTM according to the spectrum feature extraction method. Three cases of feature extraction were separately conducted: MFCC, GTCC, and both MFCC and GTCC. When LSTM and Bi-LSTM are compared, the overall performance was higher for the Bi-LSTM. As shown in Table 3, the best performance of 90.38% resulted when learning was performed by the Bi-LSTM model and the features were extracted from both the MFCC and GTCC spectrum. Figure 8 shows the relevant confusion matrix, in which emotions are fairly accurately classified at an overall high probability; however, fear has a relatively lower classification accuracy compared with the other emotion types. The second experiment examined the emotion recognition performance of the CNN-based transfer learning model using the time-frequency conversion-based 2D-spectrogram features of the Korean speech data; Table 4 presents the learning parameters of this model. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Division** & **Use** \\ \hline \multirow{3}{*}{Hardware} & CPU & Intel Core i9 10900K @ 3.70 GHz \\ \cline{2-3}  & GPU & NVIDIA GeForce RTX 2080 SUPER \\ \cline{2-3}  & RAM & 128 GB \\ \hline \multirow{3}{*}{Software} & OS & Windows 10 \\ \cline{2-3}  & Programming Language & Matlab 2022b \\ \hline \hline \end{tabular}
\end{table} 
Table 1: Experimental environment. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Deep Learning Model** & **Feature Extraction Method** & **Accuracy** \\ \hline \multirow{3}{*}{LSTM} & MFCC & 87.22\% \\ \cline{2-3}  & GTCC & 86.28\% \\ \cline{2-3}  & MFCC + GTCC & 88.47\% \\ \hline \multirow{3}{*}{Bi-LSTM} & MFCC & 89.59\% \\ \cline{2-3}  & GTCC & 87.72\% \\ \cline{2-3}  & MFCC + GTCC & 90.38\% \\ \hline \hline \end{tabular}
\end{table} 
Table 3: Accuracy of 1D spectrum feature-based LSTM and Bi-LSTM models. \begin{table}
\begin{tabular}{c c c} \hline \hline
**Training Options Parameter** & **Optimization Function** & **Gradient Threshold** & **Mini-Batch Size** & **Epoch** \\ \hline \multirow{3}{*}{Parameter values} & Adam & 1 & 512 & 5 \\ \cline{1-1} \cline{2-3}  & & & \\ \hline \hline \end{tabular}
\end{table} 
Table 4: Learning parameters of CNN transfer learning model. Table 5 presents the performance of VGGish and YAMNet, among the CNN-based transfer learning models, according to the spectrogram feature extraction method. The features were extracted for the Bark, ERB, and log-mel spectrograms for the experiment. Here, the performance of the YAMNet model is overall more outstanding compared with the VGGish model. In this experiment, the best performance of 94.91% resulted when ERB spectrogram features were extracted and learning was performed based on the YAMNet model. Figure 9 shows the relevant confusion matrix. Similar to the CNN-based model, each type of emotion was classified with an overall high probability of 90% or higher; however, fear has a relatively lower classification accuracy than the other types of emotion. Figure 8: Confusion matrix of Bi-LSTM SER model.