# Speech Emotion Recognition Based on Two-Stream Deep Learning Model Using Korean Audio Information
###### Abstract  
Identifying a person's emotions is an important element in communication. In particular, voice is a means of communication for easily and naturally expressing emotions. Speech emotion recognition technology is a crucial component of human-computer interaction (HCI), in which accurately identifying emotions is key. Therefore, this study presents a two-stream-based emotion recognition model based on bidirectional long short-term memory (Bi-LSTM) and convolutional neural networks (CNNs) using a Korean speech emotion database, and the performance is comparatively analyzed. The data used in the experiment were obtained from the Korean speech emotion recognition database built by Chosun University. Two deep learning models, Bi-LSTM and YAMNet, which is a CNN-based transfer learning model, were connected in a two-stream architecture to design an emotion recognition model. Various speech feature extraction methods and deep learning models were compared in terms of performance. Consequently, the speech emotion recognition performance of Bi-LSTM and YAMNet was 90.38% and 94.91%, respectively. However, the performance of the two-stream model was 96%, which was a minimum of 1.09% and up to 5.62% improved compared with a single model. speech emotion recognition; human-computer interaction; two-stream; bidirectional long-short term memory; convolutional neural network +
Footnote †: journal: Applied  
used with an auditory perception-based front-end using auditory signal processing and a temporal attention mechanism. In this system, the front-end is used to generate time-modulated signals, and the attention-based back end is used to identify emotional states in speech. [PERSON] [6] proposed an end-to-end model utilizing a multihead self-attention mechanism and convolutional layer that uses both encoded language and audio spectrogram as input. [PERSON] [7] proposed a deep learning-based feature fusion method for heightening the performance of an SER system. The extraction of hyperprosodic features (EHPF) involves extracting hyperparameter statistical features from prosodic features. Two-dimensional spectrogram features are extracted from the raw voice signal and then used as input for training the CNN network. Further, the spectra-based feature vectors are extracted using the CNN network and EHPF feature vectors are fused for emotion recognition via a deep neural network (DNN). The experiment confirmed that the proposed model effectively improved the SER accuracy. [PERSON] [8] proposed a 3D convolutional recurrent global neural network (CRCNN) that uses log-mels (static, deltas, and delta-deltas for the log mel-spectrogram) as input for SER. The model consists of a CNN to extract log-mels and local invariant features, followed by a recurrent neural network (RNN) that learns temporal dependency between different time-step local invariant features. Finally, the most active feature is selected using the global max pooling mechanism. To evaluate model performance, experiments were conducted on four datasets, and the effectiveness of the model was demonstrated by significantly improving the SER accuracy, compared with that of other approaches. [PERSON] [9] proposed a two-stream emotion-embedded autoencoder, which is a new type of autoencoder architecture, for extracting the features of deeper emotions. The first stream of the autoencoder is a basic autoencoder used to learn the best representation of speech. The second stream learns the information of strong emotions in labels through an emotion-embedding path. Finally, the autoencoder and emotion embedding are combined, and batch normalization (BN) is then converted to instance normalization (IN). In the emotion classification process, deep emotion features output by the two-stream autoencoder are fused with the IS10 feature set obtained from the openSMILE toolkit, and emotion classification is then performed by adopting a fully connected network based on the connected feature vectors.