# Multi-Modal Emotion Recognition Using Speech Features and Text-Embedding
## 5 Conclusions  
As intelligent personal assistants, chat-bots, and AI speakers are rapidly popularized, these approaches can be applied to human interfaces to precisely recognize the user's emotional state and to provide personalized media according to the user's emotions. They can also develop a better understanding of human emotions which could improve human-machine interaction systems. In terms of perceptual and cognitive sciences, using speech signals and text data simultaneously can be helpful in accurate and natural recognition. However, because the characteristics of the methods to recognize emotions from speech and text sequences are different, combining the two inputs is still a challenging issue in the area of emotion-recognition. Additionally, to accurately train deep learning models, considerable amounts of data are required; however, previous speech emotion recognition studies used far less data than other approaches. In this paper, we discussed an emotion recognition method which increased accuracy by using both speech and text data, utilizing the strength of each type of data. To do this, we extracted features from both speech and text data. We extracted 43 feature vectors from the speech datasets, and the probability of each emotion was deduced using these vectors through the softmax function, which consisted of a long short-term memory (LSTM) and fully connected layers. Then, after tokenizing the text data into consonants and vowels, text-embedding vectors of 256 dimensions were extracted by embedding each token using a pre-trained Tacotron encoder. The extracted  
\begin{table}
\begin{tabular}{c c c c} \hline \hline \multirow{2}{*}{**Model**} & \multicolumn{3}{c}{**Amount of Data**} \\ \cline{2-4}  & **Speech** & **Text** & **Multi-Modal** \\ \hline Yoon [26] & 89.13\% & 63.68\% & 91.35\% \\ [PERSON] [30] & 92.67\% & 65.98\% & 93.34\% \\ [PERSON] [31] & 75.26\% & 68.11\% & 81.31\% \\ Proposed & 94.86\% & 68.42\% & 95.97\% \\ \hline \hline \end{tabular}
\end{table}
Table 2: Accuracy comparison with other papers.  
embedding vectors entered the input of the LSTM and fully connected layers, and the probability of each emotion was deduced by the final softmax layer in the same way as the speech datasets. Lastly, the average values for each emotion were calculated with the results of each softmax function from the speech and text data. Then, the desired emotion was selected based on the emotion that had the highest average value. We also established a Korean language emotion speech dataset. Transcripts of scenes from media such as TV dramas or movies were categorized according to different emotions. The emotions used in the experiments fell into four categories: anger, happiness, sadness, and neutral, as used in many emotion studies. The proposed model had the highest accuracy, 95.97%, in recognizing emotions.  
The number of studies using Korean language corpora and embedding is conspicuously smaller than those using English-based natural language processing. In the future, studies using Korean text processing and embedding can be expected to combine accurate emotion recognition technology and communication interface technology by utilizing both speech and text information.  
Conceptualization, S.-W.B.; methodology, J.-H.K. and S.-P.L.; investigation, J.-H.K.; writing--original draft preparation, S.-W.B.; writing--review and editing, S.-P.L.; project administration, S.-P.L. All authors have read and agreed to the published version of the manuscript.  
This work was supported by Sangmyung University.  
Not applicable.  
Informed Consent Statement: Informed consent was obtained from all subjects involved in the study.  
Data Availability Statement: Not applicable.  
This work was supported by the 2021 Research Grant from Sangmyung University.  
The authors declare no conflict of interest.