# Speech Emotion Recognition Based on Two-Stream Deep Learning Model Using Korean Audio Information
###### Abstract
The proposed model was evaluated on the IEMOCAP database, resulting in a 71.56% classification accuracy. [PERSON] [10] proposed a new parallel network for SER by connecting a ResNet-CNN-Transformer encoder. ResNet uses the Gaussian error linear unit (GELU) as an activation function to solve the gradient vanishing problem caused by the deepening of the network, while the CNN is used to compute fewer parameters to then improve the fitting expression capability of the network. Furthermore, a transformer encoder is used to predict the frequency distribution of various emotions using a multihead self-attention layer considering the continuity of speech over time. For the fusion of these three models, the outputs of the ResNet and CNN are planarized to a one-dimensional vector and connected with the output of the transformer encoder; subsequently, the fully connected layer and softmax layer are sequentially connected to classify eight types of emotions. This model was assessed on the RAVDESS dataset and achieved an 80.89% classification accuracy, which is higher than that observed in previous studies. [PERSON] [11] proposed a model that uses a hybrid of self and multihead attention mechanisms along with dilated convolutions and an LSTM to achieve a comparable performance. Such a method computes global context dependencies between features in parallel using multihead attention. Furthermore, the global context and long-term dependency are computed using the self-attention mechanism in the Bi-LSTM layer stack. Using a dilated convolution layer improves the receptive field because the increase in the number of parameters is low, compared with the number of layers. As models that use raw signals tend to confuse happiness with anger or neutral with sadness, they typically employ spectrum and audio quality features extracted from raw audio signals as input. In terms of the performance evaluation of the proposed model, 96.36% accuracy was achieved on the EMODB dataset and 88.96% accuracy was achieved on the RAVDESS dataset. Unfortunately, previous studies have limitations in that multidimensional feature information cannot be utilized as SER is conducted using only one of the one-dimensional spectral features or two-dimensional spectrogram features. Therefore, this study designed a two-stream-based SER model involving a Bi-LSTM network and CNN-based transfer learning model using multidimensional features (1D and 2D) of the Korean speech emotion database containing eight emotions, and then comparatively analyzed the emotion recognition model's performance according to diverse feature extraction methods. Since it uses multidimensional features, it is possible to utilize multidimensional feature information of audio, which has the advantage of being able to classify a person's emotional state more objectively and accurately. The remainder of this paper is organized as follows. Section 2.1 discusses the theoretical concept of the one-dimensional spectrum feature reaction method and two-dimensional spectrogram feature extraction method of speech data. Section 2.2 introduces the deep learning models used in this study, and Section 2.3 details the architecture of the proposed two-stream-based SER model employing Bi-LSTM and CNN. Section 3 overviews the Korean speech database and then discusses the comparative analysis of the results and SER performances. Section 4 discusses the findings of this study. Finally, Section 5 explains the outcomes of this study and proposes direction for future research.